Battelle - Ohio School Boards · Battelle for Kids is utilizing visual representations of copyrighted EVAAS® Web reporting software from SAS Institute Inc. in this document for instructional

11/19/12

1

Understanding Teacher Value-Added Reports Mary Peters, Ph.D., Battelle for Kids OSBA Capital Conference – Fall 2012

© 2012, Battelle for Kids.

Session Targets

  Learn how to use teacher value-added reports for school improvement and not just evaluation.

  Understand how teacher value-added reports are created.

  Discover what you and your staff need to understand about teacher value-added reporting.

  Know how you can support teachers who receive value-added reports.


Being Future Ready in Ohio

11/19/12

2


Evidence of Student Learning

Collecting and measuring student progress and attainment. Implications:   Collecting evidence of student learning

using multiple measures, including indicators of student growth

  Developing a strong, reliable, and balanced assessment system

  Collecting formative evidence of student learning around new standards

  Preparing for new assessments


Evidence of Teacher Effectiveness

Measuring teacher success using multiple measures. Implications:   Ohio is currently working to identify and develop

multiple measures of teacher effectiveness.   Educators need useful information to help them

know if their classroom practices are effective and how they may improve.

  Measures of student growth must be determined that are: fair and equitable; sufficiently rigorous; and reliable and valid.


Backdrop: Key Statewide Deliverables of RttT

  Teacher Value-Added Reporting   30% of LEAs link in Year 1 RttT—reports received fall 2011

(primarily LEAs in Battelle for Kids’ expanded value-added reports projects along with some SIG schools)

  60% of all LEAs link in Year 2 (represents RttT LEAs)   100% of all LEAs in Ohio link in Years 3 & 4   Requires teacher linkage each spring to verify teacher

assignments and teachers’ instructional time with students

  Professional development and resources will address the use of value-added for school improvement and implications of teacher-level reporting

11/19/12

3


Why Value-Added?


Achievement and Progress


School Performance and Poverty Level – Math

Battelle for Kids is utilizing visual representations of copyrighted EVAAS® Web reporting software from SAS Institute Inc. in this document for instructional purposes.

11/19/12

4


School V-A Gains and Poverty Level – Math



School Performance and Poverty Level – Reading



School VA Gains and Poverty Level – Reading


11/19/12

5


The Power of Two: Achievement & Progress

Achievement Test Results

•  School E

Pro

gres

s O

ne Y

ear’

s G

row

th

Standard

Low Progress Low Achievement

Low Progress High Achievement

High Progress Low Achievement

High Progress High Achievement

•  School A

•  School B

•  School C

•  School D

•  School F

•  School H

•  School K

•  School G

•  School J


MAAP: Matrix of Achievement and Progress


What is a Growth Measure?

  “Growth, in its simplest form, is a comparison of the test results of a student or group of students between two points in time where a positive difference would imply growth.”

— Excerpted from Selecting Growth Measures: A Guide for

Educational Leaders, Battelle for Kids.

11/19/12

6


Spectrum of Growth Measures

  Simple comparisons of student achievement   Compare academic growth of a group of students between

two points in time   Descriptive analyses

  Begin by describing a student’s performance relative to that of peers’

  Can be aggregated   Value-added models

  Estimate the influence of schools or classrooms on academic growth rates of a group of students with statistical confidence

  Can be simple or advanced. Ohio has an advanced model.


  Tests are the most convenient way to measure students’ achievement levels, but:   All measures have error   Students don’t always score where they should

  Guessing   Cheating   Other outside factors beyond a teacher’s control

  Small numbers of students in some teachers’ classrooms make it even more difficult to produce a system that is fair

Why Is It So Difficult To Measure Growth?


  Follows individual students across time

  Uses all available student test data

  Students’ growth is compared to their own history   Estimates the school’s influence or “school effect”

on a group of students

  Growth expectations are a policy decision and can be fixed (pre-determined standard) or normative (compared to the pool)

  Statistical models can accommodate various testing regimens

Ohio’s EVAAS®: Value-Added Analysis Overview

11/19/12

7


  Tests must possess the following three properties:   They must be highly correlated to curricular

objectives   They must have sufficient stretch to differentiate

student achievement levels at both the lower and higher ends

  They must be sufficiently reliable

Required Test Properties for Value-Added Analysis


EVAAS® Information Used for Diagnostic Purposes   Value-added measure

  Group statistic – measures the impact schools and teachers have on a group of students

  It’s about us, the adults   Tells us: Is this “program” working? For whom?   It’s about the past – fall report release represents the

effects of the program the prior school year   Projection information

  It’s about individual students   Tells us: To what extent are students on a positive

trajectory?   It’s about the future – probability of future success


What Do You See In Value-Added Reports?

  Standard error calculations   Establishes confidence band or range of values plus or

minus the most likely value   Scores represented on Normal Curve Equivalency

scale versus scaled scores or percentiles   Provides equal intervals from 1 to 99   Allows for averaging over time

11/19/12

8


Extended Value-Added Reporting Uses Different Approach

  The analysis used for the Ohio value-added results in grades 4-8 math and reading can only be used when tests are uniformly administered in consecutive grades

  A different analysis must be used when tests are not given in consecutive grades   Predicted mean approach   Requires a minimum of three prior student data

points (across subjects)   Normalized—annual comparison is to the pool


Battelle for Kids is utilizing visual representations of copyrighted EVAAS® Web reporting software from SAS Institute, Inc. in this presentation for instructional purposes.

Value-Added Reports


Key Reports School Value-Added (MRM)

11/19/12

9


Key Reports School Diagnostic (MRM)


 What reports do teachers have

access to?  How can they interpret these data?

Teacher Value-Added Reports


Ohio Teacher Evaluation System (OTES)

http://www.youtube.com/ohioeddept

Source: Ohio Department of Education

11/19/12

10


Student Growth and Teacher Value-Added


Using Multiple Measures Correctly

  All measures should inform practice

  Multiple measures should clarify, not confuse

  More measures are not necessarily better

  Strong measures should “count” the most


Remember, All Measures Are Estimates

  No measure is perfect, including student growth.   Observations of teacher performance are also

imperfect estimates.   Educators need to continuously improve all

measures by how:   We administer them,   We build our confidence that they are reliable

estimates, and   We use them to improve practice.

11/19/12

11


What is Link/Roster Verification?

  Process of accurately linking students to teachers for the purpose of ensuring accuracy of value-added reports

  It provides reliable data that reflects what is actually taking place in the classroom, capturing the instruction that each teacher provides over the course of a school year

  Link/Roster Verification training will be available in the winter and spring


Teacher-Level Value-Added Report: Aggregate Level Reporting

Aggregate-level effectiveness of this teacher over time

The table also displays how this teacher performed compared to the state’s three-year average and the district’s three-year average in that same grade level and subject area (most recent year first).


Teacher Value-Added Diagnostic Report: Disaggregate-Level Report This teacher’s relative effectiveness with her three prior achievement subgroups

11/19/12

12


Sample Student Pattern Report


Responding to Classroom- Level Value-Added Information How can I use this information to:   Accelerate student growth?   Help teachers grow professionally?   To inform school improvement?


Leadership Practices Matter

  Leadership practices that contribute to better instruction:

1. Focusing on goals and expectations for student achievement

2. Creating structures and opportunities for teacher collaboration

3. Attending to teachers’ professional development needs

Source: Wahlstrom K., et al., 2010

11/19/12

13


1. Support a Healthy Culture

How is your leadership team preparing to:   Be sensitive to the impact teacher reports have on a

teacher’s sense of self-worth?

  Build an environment where teacher-level data can be routinely shared openly among colleagues in a manner that is safe and respectful?

  Help teachers respond to the information they receive by guiding them toward reflective thinking and taking productive action?


2. Initiate Strength-Based Conversations

 How can you leverage effective practices?   What patterns of strength do you see in your data?

 How can you leverage your strengths?   With which student achievement subgroups are you

producing the most growth? Why?  How can you leverage the strengths of your team?

  Who is producing the most overall growth?   With which student achievement groups?   What can you learn from each other?


3. Align This Work To:

  Teacher-Based Team processes   Monitoring student progress   Ongoing data-based decision making

  Ohio Teacher Evaluation System   Professional growth plans   Evidence of student growth

  Formative Instructional Practices   Effective pedagogy   Monitoring student progress

11/19/12

14


A Systems Approach

Increased Student

Achievement

Ohio Standards

for the Teaching

Profession

Formative Instructional

Practices

Ohio Standards

for Principals

Teacher-Based Team

Process


Leaders Support Teacher-Based Teams

www.ohioleadership.org


Ohio Standards for the Teaching Profession

11/19/12

15


4. Differentiate Support for Teachers

Source: Ohio Teacher Evaluation System Model, Ohio Department of Education, pg. 11, May 2012


Leaders Support High Quality Instruction: Formative Instructional Practices

  In order to move teachers to higher levels of observed practice, professional learning should be aimed at moving teachers’ instructional competencies forward.


FIP Your School® Ohio

11/19/12

16


Additional Support


Take Advantage of Ohio’s Value-Added Network of Support: VALs & DVALs   Value-Added Leaders (VALs): 90 VALs who support district/

community school value-added teams   District Value-Added Leaders (DVALs): On average, 3–5 person

team from districts/community schools who provide support to principals and teachers in the use of value-added information

1.   Go to the Ohio Student Progress Portal, www.BattelleforKids.org/Ohio

2.   Choose “Value-Added Network of Support” from Quicklinks

3.   Utilize “Find your VAL/DVAL” feature to contact your local support system.


Helpful Tools for Leaders and Staff   EVAAS® Interactive Site – Diagnostic Reports and Help Menu   BFK Ohio Student Progress Portal, www.BattelleforKids.org/Ohio   Online Courses via Ohio•Learn: value-added and FIP   Focus Guides, A system of continuous improvement   Understanding & Using Value-Added Analysis Toolkit   VA Book: How to Use Value-Added to Improve Student Learning

11/19/12

17

www.BattelleforKids.org http://twitter.com/BattelleforKids

Graduate Course Credit Available Introducing “Read 2 Earn”

Through our relationship with California Lutheran University, an NCATE-accredited institution, educators can now receive one semester hour of graduate course credit for reading any Corwin book. Yes, that’s right: any Corwin book can now be used in an independent study course to earn one semester hour of graduate credit.

This gives busy educators who need graduate credit to take the next step in their careers the opportunity to earn credit at their own pace through independent study. Simply select a Corwin book, register for credit at www.corwin.com/read2earn, read the book, and complete three writing assignments within three months.

Best of all, this option only costs $199, which is affordable for educators on a budget.

Start spreading the word today! Read 2 Earn can help a teacher or administrator earn graduate credit from an accredited university.

Visit www.corwin.com/read2earn for more information.

RttTAchieveGraduateSucceed

VALUE-ADDED ONLINE COURSES BY LE ARNING PATH

Building-level Value-Added Learning Path

For Math and Reading

MG 1002Introduction to the Building-level

Value-Added Learning Path

VA 1010Introduction to Value-Added

Progress Metrics

VA 1020Progress and Achievement

MG 1040From Macro to Micro: Examining

Building-Level Value-Added Reports

MG 1060 Interpreting School

Value-Added Reports

MG 1070Interpreting Value-Added

Summary Reports

MG 1090Interpreting School Diagnostic Reports

MG 1100Interpreting Diagnostic Summary Reports

MG 1110 How to Use the School Search

MG 1120 How to Use the Student Search

MG 1130Interpreting Individual Student Reports

MG 1140Interpreting Teacher-Level

Value-Added Reports

MG 1150Providing Access to

Value-Added Reports

12 hours*

District- or System-level Value-Added Learning Path

For Math and Reading

MG 1001Introduction to the District-level



Progress Metrics



System-Level Value-Added Reports

MG 1050 Interpreting System

Value-Added Reports

MG 1070Interpreting Value-Added

Summary Reports

MG 1080 Interpreting System Diagnostic Reports

MG 1100Interpreting Diagnostic Summary Reports

MG 1110 How to Use the School Search

MG 1150Providing Access to

Value-Added Reports

9 hours*

Teacher-Level Value-Added Learning PathFor Science, Social Studies

and High School

MG 1003Introduction to the Teacher-level



Progress Metrics



Building-Level Value-Added Reports

MG 1060 Interpreting School

Value-Added Reports

MG 1090Interpreting School Diagnostic Reports

MG 1120 How to Use the Student Search

MG 1130Interpreting Individual Student Reports

MG 1140Interpreting Teacher-Level

Value-Added Reports

8 hours*

Building- and Teacher Level Value-Added Learning Path

For Science, Social Studies and High School

PM106Introducing Value-Added Reports

PM112Logging In, Examining the Home Page, and Navigating Value-Added Reports

PM115 The Predicted Mean Approach to School

Value-Added Reports

PM117Interpreting School Diagnostic Reports

PM120Interpreting Diagnostic Summary Reports

PM121Interpreting Individual Student Reports

PM123Performing Searches and Creating Custom Reports

PM128 Interpreting Teacher-Level

Value-Added Reports

8 hours*

Organized into four role-based learning paths, the following interactive online courses will help educators to further understand, interpret and use EVAAS® value-added analysis to inform practices and accelerate student progress. The courses are designed for educators to work independently or with a professional learning team. Since many of the courses in these paths overlap, all four series could be completed in approximately 24 hours.

A suite of updated value-added online courses is now available at no cost to all Ohio K–12 public schools as part of Race to the Top.

To enroll, visit the Ohio Student Progress Portal at www.BattelleforKids.org/Ohio. Log in to “My Portal” and select “My Learning”. For more detailed instructions, select “New BFK-LearnTM User Guide”.


Course offered in the District-level Value-Added Learning PathCourse offered in the Building-level Value-Added Learning PathCourse offered in the Teacher-Level Value-Added Learning PathVA: General value-added coursesMG: Courses featuring value-added reports using the SAS EVAAS® Multivariate Response Model (MRM), i.e. mean gain approach.PM: Courses featuring value-added reports using the SAS EVAAS® Univariate Response Model (URM), i.e. predicted mean approach.

* Approximate amount of time to complete learning path


®

FORMATIVE INS TRUC TIONAL PR AC TICES ( F IP) ONLINE COURSES

Through Race to the Top, Ohio educators have access to free formative instructional practices online learning modules designed to help teachers and leaders advance their practice. Learning paths include:

• Foundations of Formative Instructional Practices which consists of five modules designed to build foundational understanding of the core components of FIP.

• Leading and Coaching Formative Instructional Practices is designed to help school administrators, coaches, and others in leadership or coaching roles learn how to ensure a successful and systemic implementation of FIP.

• Creating Clear Learning Targets (two paths) in English language arts (ELA) and social studies are designed to help educators transition to Ohio’s New Learning Standards. Teachers, curriculum directors, instructional coaches, and others working on new standards can use these grade- and subject-specific modules to learn how to deconstruct standards, create and classify learning targets, and organize the targets into logical progressions for learning.

Visit www.FIPYourSchoolOhio.org to learn more about these modules, associated facilitation guides, and professional development opportunities.

Foundations of Formative Instructional Practices Learning Path

FP0001Module 1: Introduction to Formative

Instructional Practices

FP0002Module 2: Clear Learning Targets

FP0003Module 3: Analyzing Evidence and

Providing Effective Feedback

FP0004Module 4: Analyzing Evidence and

Providing Effective Feedback

FP0005Module 5: Student Ownership of Learning:

Peer Feedback, Self-Assessment, and More

Leading and Coaching Formative Instructional Practices Learning Path

FP0006Module 6: Leading Formative


FP0007Module 7: Coaching Formative


Creating Clear Learning Targets for ELALearning Path

FP0008Creating Clear Learning Targets for ELA in

Elementary School


Middle School


High School

Creating Clear Learning Targets for Social Studies Learning Path

FP0011Creating Clear Learning Targets for Social

Studies in Elementary School


Studies in Middle School


Studies in High School

HELPING EDUCATORS DO THEIR WORK BETTER

2455 Teller Road, Thousand Oaks, CA 91320 • Ph: 800-233-9936 • Fax: 800-417-2466 • www.corwin.com

How to Use Value-Added Analysis toImprove Student Learning A Field Guide for School and District Leaders

Kate Kennedy, Mary Peters, Mike Thomas

A step-by-step guide to transforming student learningwith value-added analysis

Value-added analysis is the most robust, statistically significantmethod for helping educators see student growth over time. This book provides a field-tested continuous improvement model for using value-added information to increase student learning in boththe classroom and schoolwide. The five-step process shows how to:

Create the conditions for success Examine district, school, and classroom reports to assess strengths and challenges Use these reports to create an improvement plan Implement instructional changes Evaluate and adjust the changes as the new school year starts

Table of Contents List of Figures / Acknowledgments / About the Authors / Foreword / Preface / 1. Step I : What is Value-Added Analysis? / 2. Step I: Jump into Value-Added Analysis / 3. Step I: The Framework for Systemic Improvement / 4. Step II: Assess District-Level Value-Added Reports: Using Results to Determine Strengths and Challenges / 5. Step II: Assess Building-Level Value-Added Reports: Using Results to Determine Strengths and Challenges / 6. Step II: Assess Teacher-Level Value-Added Reports: Using Results to Determine Strengths and Challenges / 7. Step III and IV: Identify Root Causes and Produce an Improvement Plan / 8. Step V: Take Action, Monitor, and Adjust / Conclusion / Glossary / References / Index

December 2011, 200 pages, 7" x 10" Paperback: $31.95, D11863-978-1-4129-9633-4

Payment MethodCheck # _________________ Payable to Corwin

Email:________________________________________

In case we have questions...

Qty. Book# Title

All orders are shipped Ground Parcel.For other shipping methods and costs, call (800) 233-9936

Signature: ______________________________________

Please send me your latest catalog. Free

Please attach original purchase order.

Yes, you may email other Corwin offers to me.Your email address will NOT be released to any third party.

BILL TO (if different)

Purchase Order #______________________________

Name:__________________________________________

Title:___________________________________________

Organization:____________________________________

Address:________________________________________

City:____________________________State:___________

Zip Code:________________________________________TelephoneRequired:

Four EASY WAYS to order!

Priority Code:

Fax:

Telephone:Credit Card #:

SHIP TO

Name:__________________________________________

Title:___________________________________________

Organization:____________________________________

Address:________________________________________

City:___________________________State:____________

Zip Code:_______________________________________

DISCOUNTS ARE AVAILABLE for large quantity orders —

CALL (800) 831-6640and ask for a sales manager.

Prices subject to change without notice.Professional books may be tax-deductible.

Federal ID Number 77-0260369

ORDER FORM

Expiration Date

month year/

3 or 4 digit code from back (VISA, MC, Discover)or front (AMEX) of card

Security Code

CREDIT CARD

Unit Price Total Price

Total Book Order

Sales TaxAdd appropriate sales tax in AL, AZ, CA, CO, CT, DC, FL, GA, IL, IN, MD, MA, MN,NC, NJ, NY, OH, PA, RI, TX, VA, VT, WA. (Add appropriate GST & HST in Canada)

Shipping and Handling$5.95 for first book, $1.00 each additional book

Canada: $11.95 for first book, $2.00 each additional book

Total Amount Due $Remit in U.S. dollars

SATISFACTIONGUARANTEEDor Your Money Back!

(800) 831-6640 Press 4CorwinPressSpeakers.com

Ask About Our

Speakers’ Bureau Before Planning Your Next Event!

New ProductE-Alerts

Receive a monthly email aboutnew Corwin productsprior to their release

Sign Up Today at www.corwin.com/alerts

D11863

D11863-978-1-4129-9633-4 Paperback $31.95

“This book offers a practical, engaging introduction to value-added assessment. It should be read by educators at all levels. The authors demystify a complex topic and give educators the tools they need to use value-added data to help students learn.”—Bob Taft, Former Governor of OhioDistinguished Research Associate, University of Dayton

“Value-added assessment is an extraordinarily important breakthrough in education research that provides a direct empirical measure of instructional effectiveness at the classroom level. This book could not be more timely, given the sea change now underway in how teachers and administrators are being evaluated and compensated.”—Theodore Hershberg, Professor, Public Policy & HistoryCenter for Greater Philadelphia, University of Pennsylvania

“A welcome, plain-spoken, and eminently practical guide to making the most of value-added analysis to strengthen data-driven decision making and boost achievement in our schools.”—Chester E. Finn, Jr., PresidentThomas B. Fordham Institute

Documents

Battelle - Ohio School Boards · Battelle for Kids is utilizing visual representations of copyrighted EVAAS® Web reporting software from SAS Institute Inc. in this document for instructional