Oerc va sessions dec 12 2013 final for post

© 2013, Battelle for Kids. All Rights Reserved.

VALUE-ADDED INFORMATION

IN PRACTICE

Jamie Meade, Managing Director,

Strategic Measures, Battelle for Kids

December 12, 2013


Session Objectives

Provide value-added information relevant to

the practitioner’s perspective

What should teachers and principals know and

understand about SAS® EVAAS® value-added

reporting?

How may educators use SAS EVAAS value-

added information to inform practice and

impact student academic progress?


Value-Added Measures in Ohio

For more than 10 years, Battelle for Kids has

provided support for professional learning and

fostered collegial dialogue for understanding

and using value-added measures.

Battelle for Kids continues to advocate for the

use of value-added measures to inform

practices and support student learning.


Ohio’s Value-Added Models

Generated by SAS® Analytics: Customers in 135 countries

More than 65,000 business, government

and university sites

SAS customers or their affiliates

represent 90 of the top 100 companies

on the 2012 FORTUNE Global 500® list


Value-Added Information in Practice:

Building Awareness

Understanding the Difference

Achievement Measures

Progress/Growth Measures


Achievement and Growth:

Understanding the Difference

Achievement

Growth


Measuring Growth is Important for

ALL Students

Proficient

Grade

3rd

4th

5th

6th

7th

8th


Why can’t we simply compare OAA scaled scores

from one year to the next to measure growth?

Level 3rd

4th

5th

6th

7th

8th

Advanced 447-521 452-551 439-548 448-575 458-574 459-537

Accelerated 429-446 432-451 424-438 429-447 436-457 432-458

Proficient 400-428 400-431 400-423 400-428 400-435 400-431

Basic 378-399 377-399 382-399 378-399 378-399 379-399

Limited 240-377 239-376 245-381 235-377 268-377 288-378

Spring 2013 OAA Math Scaled Score Ranges


Value-Added Information in Practice:

SAS EVAAS MRM Model

Mutlivariate Response Model (MRM) Grades 4–8 Reading and Math

Ohio Achievement Assessments


SAS®

EVAAS®

Value-Added:

Two Models in Ohio

Mean Gain Approach (MRM)

4–8 Reading and Math (OAA) Reports

Predicted Mean Approach (URM)

5 & 8 Science (OAA) School and District Reports

Extended Testing

ODE RttT Mini-Grant

BFK SOAR


SAS®

EVAAS®

MRM Model*

Uses grades 3–8 Reading and Math OAAs

Compares the average growth of students in

the most recent year to the average growth of

students in 2010 (state’s baseline year).*

Growth expectation is defined as maintaining

placement in the distribution of NCE scores

from one year to the next.*

*conceptual level


Raw Score Scaled Score NCE

Sample

Raw Score

Range

Sample

Scaled Score

Range

Normal Curve

Equivalent

NCE

52

0

551

247

99

1


Scaled Scores Converted to NCEs in

2010 Baseline Year

Example

2010 Scaled Scores

Rank Ordered

551

247

Normal Curve Equivalent (NCE)

99

1

Conversion Values Are Fixed/Frozen


Value-Added Terminology


The NCE is similar to a percentile rank in that scores are derived from scaled scores and ranked based upon performance.

A significant difference between percentile rank and NCE is that an NCE scale is an equal interval scale.



1 10 20 30 40 50 60 70 80 90 99

1 5 10 20 30 40 50 60 70 80 90 95 99

Distribution

of Scores

Normal Curve

Equivalents

PercentileEquivalents



Baseline Score

Group of students’ prior year mean NCE

Example: Spring 2012 OAA mean NCE

Observed Score

Group of students’ new/most recent mean NCE

Example: Spring 2013 OAA mean NCE


Basic, Conceptual Example

Scaled scores are converted to NCEs

394 = 46 Student 1 430 = 59

402 = 50 Student 2 417 = 54

384 = 42 Student 3 400 = 49

394 = 46 Student 4 390 = 44

410 = 52 Student 5 425 = 57

Mean Baseline = 47.2

Grade 6

Baseline

Grade 7

Observed

Mean Observed = 52.6

A basic measure of the growth for this group is 5.4 NCEs

Growth = Mean Observed – Mean Baseline

Growth = 52.6 – 47.2 = 5.4 (Mean NCE Gain)


Teacher Value-Added Report

Note: Battelle for Kids is utilizing visual representations of

copyrighted EVAAS® Web reporting software from SAS in this

presentation for instructional purposes.


Statistical Terminology

Standard ErrorAll measures of student learning contain error.

Standard error is a measure of the

uncertainty.

In the EVAAS teacher value-added report, the

size of the standard error is influenced by

N size (size of the student group)

Missing scores



Growth Index

Since the size of the standard error (degree of

certainty) will vary across teachers, their

estimated gain must be standardized to include

both the estimate and the degree of certainty

(standard error).

Divides a teacher’s estimated gain by the

associated standard error.


Teacher Value-Added

Progress Table


Levels of Teacher Value-Added Effects

Most

Effective

Teacher's index: 2 or greater

Students are making substantially more progress than the state

growth standard.

Above

Average

Teacher's index: equal to or greater than 1, but less than 2

Students are making more progress than the state growth

standard.

Average

Teacher's index: equal to or greater than -1, but less than 1

Students are making the same amount of progress as the state

growth standard.

Approaching

Average

Teacher's index: equal to or greater than -2, but less than -1

Students are making less progress than the state growth standard.

Least

Effective

Teacher's index: less than -2

Students are making substantially less progress than the state

growth standard.


Teacher Diagnostic Graph


Key Considerations:

Systemic Programs, Delivery Models, Structures,

Services, etc.

Access to Professional Learning

Curriculum

Instruction

Assessment

Using Value-Added to Inform Practice


Link/Roster Verification Process

Ohio’s Journey:

Spring

2012

60%

of Ohio’s

LEAs

Participated

Spring

2011

30%

of Ohio’s

LEAs

Participated

Spring

2013

100%

of Ohio’s

LEAs

Participated


What is Roster Verification?

A process to capture the instructional linkage

between teachers and students.

Teachers verify three questions for each class

roster:

1. I taught these students

2. During these months

3. For __% of instruction (sole or shared)


Why Do We Link?

Student schedule files do not always capture

an accurate connection between teachers and

the students they teach.

Roster Verification provides teachers and

principals with a solution to review, modify,

and verify class rosters to ensure the

instructional linkage between teachers and

students is accurate.

@BattelleforKids

facebook.com/battelleforkidsorg

youtube.com/battelleforkids

BattelleforKids.org

Jamie Meade

[email protected]

Thank You!

mailto:[email protected]

Education

Oerc va sessions dec 12 2013 final for post