Exploring Alternate AYP Designs for Assessment

and Accountability Systems

1

Dr. J.P. Beaudoin, CEO, Research in Action, Inc.Dr. Patricia Abeyta, Bureau of Indian Education

NIEA 2008 Annual ConferenceSeattle, Washington

Session Goals

2

• Frame the technical and resource considerations for decision-makers

• Provide clarification on the

regulatory provisions associated with NCLB

• Share assessment and accountability design features

Guiding Questions

3

urpose

How does the purpose for the change/action serve Native children?

irection What direction will the change/action

provide the educational community?

otivation To what degree will the change/action

motivate the Native peoples?

30,000ft. SYSTEM VIEW

4

School System Components

5

Academic Content Standards“What do students need to know and be able to demonstrate?”

• Instruction “How is content organized, provided, modeled, and expanded upon?”

• Assessment “To what degree is the content expressed accurately by the leaner?”

• Accountability “Which learning indicators are being reached, showing progress, or remaining unchanged?”

Integration

6

Things to Consider…

• Changes to one component will have some influence on the remaining components

• Improvements in one area often result in benefits throughout the system

• Isolated changes create bulkization, inefficiencies, and internal stressors

7

Leverage

Compliance

If you change the academic content standards used for AYP…

If you change the assessments used for AYP…

If you change how AYP is defined…

8

Then you must provide evidence that addresses Section 1 of the USDE’s Peer Review Guidance

Then you must provide evidence that addresses Sections 2-7 of the USDE’s Peer Review Guidance

Then you must create an AYP Workbook that address 10 accountability principles.

Overview: Standards and Assessments Peer Review

9

Conditioned on what is changed, evidence must be presented to the BIE and USDE Peer Reviewers…

Section 1-Content Standards

Section 2-Achievement Standards

Section 3-Coherent Assessment System

Section 4-Technical Quality

Section 5-Alignment

Section 6-Inclusion

Section 7-Reporting

Overview: AYP Workbook

10

Principle 1-Includes all schools and districts

Principle 2-Includes all students

Principle 3-Includes an allowable method for AYP determinations

Principle 4-Includes annual accountability decisions

Principle 5-Includes subgroup accountability

Principle 6-Based on academic assessments

Principle 7-Has additional indicators

Principle 8-Separate decision for reading and mathematics

Principle 9-System validity and reliability

Principle 10-Includes participation rates

Conditioned on what is changed, policy and empirical evidence must be submitted for review by the BIE and USDE…

11

Overview: Review Process

ASSESSMENT

12

Generic Assessment Life Cycle

13

Foundational Components

Academic content standards Articulate what student are expected to

know and be able to do Grade level expectations in reading/ELA

and mathematics Grade span clusters in science Contain skill and process knowledge

14

Foundational Components

Academic achievement standards Performance level descriptors No less than three achievement levels

(with applicable labels) “Cut-Score” established using a

technically valid standard-setting approach (e.g, Modified Bookmark, Body of Work, Teacher Judgment)

Established by subject matter experts (teachers) and other panelists with applicable skills

15

Foundational Components

Assessments aligned to intended purposes

Assessments in grades 3-8 and high school: Reading/ELA Mathematics Science (3 grade clusters)

Standardized administration and scoring procedures

16

Foundational Process

Typically 18-24 month development process

Documented evidence Reviewed by external (to USDE)

experts Evaluated evidence against Peer

Review Guidance Resubmissions typical

17

USDE PEER REVIEW GUIDANCE

Standards and Assessments

18

Section 1: Academic Content Standards

Students know and are able to…

Content standards are:

Adopted in mathematics; reading/ELA Adopted in science (grade spans) Coherent and rigorous Developed by stakeholders

19

Section 2: Academic Achievement Standards

The level students have mastered the content is…

Achievement standards are: Adopted in mathematics, reading/ELA, and

science Assigned achievement levels and content-

based descriptors Established using an appropriate standard-

setting procedure Aligned to the content standards

20

Section 3: Coherent Assessment System

The different assessment components within the larger assessment system contribute…

A coherent system: Is based on state and/or local assessments Provides integrated achievement data Aggregates only comparable measures Utilizes multiple measures focused on

higher order thinking skills Implements alternate assessments

21

Section 4: Technical QualityThe technical quality of the assessments is…

High quality assessments are: Valid measures based on their purpose Constructed to measure the content standards Accurate and consistent Fair and accessible Consistent across different forms and events Standardized in their administration and

scoring

22

Section 5: Alignment

The assessments reflect the content standards characteristics by…

Strong alignment is provided by using:• Using procedures to improve alignment

• Techniques to match the content standards and patterns of emphasis

• Capture the range and depth of knowledge in the standards

• Pattern of emphasis

• Approaches to measure both content and process knowledge

• Content-based performance descriptors

23

Section 6: InclusionThe assessment system includes all students

by…

Students participate in the assessments as demonstrate by:

Impact data showing high rates of student participation

Affording students with disabilities participation under accommodated conditions

Accommodating students learning to speak English

Including migrant and other highly mobile students

24

Section 7: ReportingThe assessment reports provide accurate and

timely information about student achievement by…

Student achievement data is: Reported using valid and credible procedures Based on all students from valid assessments Reported at the individual student, subgroup,

school, district, and state-levels Disseminated in a timely manner Easily understood by parents and the public

25

USDE REVIEW GUIDANCEAYP Workbook

26

Principal 1: Includes All Schools and Districts

Includes all schools and districts Holds all schools to the same

criteria Incorporates the academic

achievement standards Provides information in a timely

manner Includes report cards Includes rewards and sanctions

27

Principal 2: Includes All Students

Includes all students Consistently defines full

academic year (FAY) Includes mobile students

28

Principal 3: Method of AYP Determinations

Applies universal proficiency by 2013-2014 as the long-term goal

Uses a method to make subgroup, school, and district AYP determinations

Establishes a starting point with statewide annual measurable objectives (AMOs)

Establishes intermediate goals29

Principal 4: Includes Annual Decisions

Determines annually the progress of schools and districts

30

Principal 5: Includes Subgroup Accountability

Includes all the required student subgroups

Holds schools and LEAs accountable for the progress of student subgroups

Includes students with disabilities (SWD) and limited English proficient (LEP) students

Includes an established minimum n-count

Includes safeguards to protect student privacy

31

Principal 6: Based on Academic Assessments

Based primarily on academic assessments

32

Principal 7: Includes Additional Indicators

Includes graduate rate for high schools

Includes an additional academic indicators for elementary and middle schools

Establishes valid and reliable additional indicators

33

Principal 8: Includes Separated Decisions for Reading/ELA and Math

Holds students, schools, and districts separately accountable for reading/ELA and mathematics

34

Principal 9: System Validity and Reliability

Produces reliable decisions Produces valid decisions Addresses changes in

assessments and student populations

35

Principal 10: Includes Participation Rates

Includes a way to calculate the rate of participation on the statewide assessments

Applies the 95% participation criteria to student subgroups and small schools

36

37

Research in Action, Inc.

World-Class Educational Quality™

Voice: 1.225.571.2408 E-Mail: ria2001@eatel.netFax: 1.225.644.8472 Website: www.ria2001.org