E & R Assessments 2007 1

The MAAECF: Assessment Procedures, Supporting Data, & Technical Considerations

Stephen N. Elliott Andrew T. Roach Vanderbilt University Georgia State University

Presented at the 8th Annual Maryland ConferenceOn Alternate Assessment

E & R Assessments 2007 2

Key Questions to be Addressed

1. What does the Mississippi Alternate Assessment of Extended Curriculum Frameworks (MAAECF) measure?

2. How do you conduct an alternate assessment using the MAAECF Rating Scale for students with significant cognitive disabilities?

3. What evidence is there to support the claim that the MAAECF yields reliable and valid results?

4. What are salient technical issues associated with this approach to alternate assessment?

E & R Assessments 2007 3

Technical Issues to Consider

Issue #1. Teachers as reliable judges of student performance.

Issue #2. Reliability of teachers’ ratings and the concept of independence.

Issue #3. Use of standards-based IEPs to drive the development and collection of evidence for a meaningful sample of items.

Issue #4. Amount and quality of classroom evidence needed for valid ratings of proficiency.

E & R Assessments 2007 4

Validity Evidence Plan

E & R Assessments 2007 5

MAAECF : A Comprehensive, Evidence-Based

Rating Scale that Uses Teachers’ Judgments to Measure Student Achievement

E & R Assessments 2007 6

Assessment of the Mississippi Extended Curriculum Frameworks (MECF)

Extended Content Standards - General statements that describe what students should understand and be able to do in reading/ language arts, mathematics, and science

Competencies and Objectives - Specific statements of expected knowledge and skills necessary to meet a content standard requirement

E & R Assessments 2007 7

MECF Organizational Structure

E & R Assessments 2007 8

LA Strands and Competencies

E & R Assessments 2007 9

MECF Classroom Tasks/Activities to Help Teach and Generate Evidence

E & R Assessments 2007 10

Sample MAAECF Items and Rating Scale Format

E & R Assessments 2007 11

Essential Features of Evidence-Based Judgment Systems like the MAAECF

Unbiased items.

High-quality evidence about students’ knowledge & skills.

Clear and objective scoring criteria.

Procedures to ensure the reliability of the scores.

Meaningful methods for communicating results of the assessment.

Trained judges – all educators using the MAAECF must attend a workshop + pass a qualification test!

E & R Assessments 2007 12

MAAECF Honors “Teachers as Tests”

The research-based concept of teachers as tests emphasizesthat teachers collect a substantialamount of data about students and when they are provided astructure for collecting evidenceand a method for quantifying andreporting this information, it canbe used like other highly reliabletest results.

Key studies on the reliability of

teachers’ judgments: Hoge & Coladarci (1989) Demary & Elliott (1998) Hurwitz, Elliott, & Braden (2007)

E & R Assessments 2007 13

What Does the MAAECF measure?

The MAAECF (regardless of level) includes more than 100 items representing prerequisite knowledge and skills in core academic areas. Rating scale items are based on the objectives specified in the Mississippi Extended Curriculum Frameworks (MECF). Note that various levels of the MAAECF exist for students in the grade clusters of 3-5, 6-8, and 12.

Teacher rates proficiency using a 4-point scale (0 = Non-Existent to 3 = Accomplished). The same item level rubric is used across all grade clusters.

Results are reported as performance levels (Minimal, Basic, Proficient, or Advanced) in each content area. Definitions and cut scores for these levels vary by grade. The results are on a continuum that is very similar to the general education test’s performance levels.

E & R Assessments 2007 14

Item Count Per Scales & Grade Levels

E & R Assessments 2007 15

Mississippi Alternate Assessment and Accountability System

E & R Assessments 2007 16

MAAECF : Administration & Scoring Procedures

E & R Assessments 2007 17

The MAAECF Process Once the IEP Team decides that a student iseligible for an alternate assessment, a 5-stepprocess must be followed:

Step 1: Align MAAECF items with IEP goals, objectives, or other learning objectives.

Step 2: Collect performance evidence for a sample of items for each strand

using Evidence Worksheets.

Step 3: Analyze and rate proficiency of all items.

E & R Assessments 2007 18

The MAAECF Process (continued)

Step 4: Summarize proficiency scores and performance level decisions. A second educator completes the process by reviewing collected evidence, providing ratings on only items with evidence samples, and checking the accuracy of the performance level decisions.

Step 5: Report results after the first rater and the individual who completes the reliability check have reached agreement.

E & R Assessments 2007 19

Team Work for Reliable Results:Summarizing the Roles of Raters 1 and 2

Rater 1

Step 1 Align Items

& Goals

Step 2Collect

Evidence for Aligned

Items

Step 3Rate All Items

Step 4Summarize

Ratings& Proficiency

Decisions

Step 5Report ReliableResults

Reliability CheckerReview Evidence

Review & Rate Items with Evidence Double Check Total Scores

Make an Independent Proficiency DecisionDetermine Agreement, Resolve Disagreements

Rater 2

E & R Assessments 2007 20

Step 1: Alignment of Items

Determine in which content areas a student will be assessed.

Identify MAAECF items that align with the student’s IEP goals and objectives, or other learning goals.

If there is not at least 1 IEP-aligned item for each competency (4 in LA, 5 in Math, and 4 in Science), then an item must be selected and evidence generated for that item.

E & R Assessments 2007 21

Step 2: Collect Evidence of Skills

Teachers must collect evidence from 2

different categories for at least 1 item for

each competency. Work Samples Tests Observations Interviews Video/Photo Audio Tape

E & R Assessments 2007 22

Characteristics of Good Evidence

Recent (collected during the current school year & dated)

Representative (several forms that are typical performances of knowledge and skills with classroom materials and instructional accommodations)

Relevant (evidence that is clearly indicative of an item that is aligned with content standards, and that identifies the relevant item number(s) for the second rater and others)

Reliable (increases the likelihood that 2 or more raters have the same understanding of the item by documenting the support provided to get the response from the student and indicating the overall accuracy of the student’s typical response)

E & R Assessments 2007 23

Evidence Collection Worksheets

Number of settings is

important to determine

generalizability of skill

Information about

support is important

Dates are needed to

determine if evidence is

recent

Description of accuracy

of the student’s

response is essential

E & R Assessments 2007 24

Quality Evidence Matters: Collected Evidence Should Meet or Exceed the Standard!

E & R Assessments 2007 25

Step 3: Criteria for Rating All Items

0 = NON-EXISTENT (Can’t do currently)

1 = EMERGING (Aware and starting to do)

2 = PROGRESSING (Can do partially and inconsistently)

3 = ACCOMPLISHED (Can do well and consistently)

***Examine descriptions of these Proficiency Rating Levels***

E & R Assessments 2007 26

MAAECF Item Level Proficiency Rating Scale and Descriptive Criteria

E & R Assessments 2007 27

Step 4: Overall Proficiency Scores and Performance Continua

Students earn an Individualized Proficiency Total Score for each content area. The total scores are used to guide the determination of which of the four Overall Performance Levels best describes the student’s achievement.

Minimal Basic Proficient Advanced

***Examine detailed Performance Levels***

E & R Assessments 2007 28

LA Performance Level Descriptor(with Cut Scores for Grades 3, 4, and 5)

E & R Assessments 2007 29

MAAECF Item Count & Score Ranges by Performance Level & Grade Clusters

E & R Assessments 2007 30

2nd Rater’s Responsibilities Review evidence of student’s knowledge and skills and then

independently rate items with which they are aligned. This is done on a separate copy of the MAAECF.

Review the 1st rater’s ratings of evidence-based items to determine general degree of agreement and then examine all item ratings to learn more about the student. These ratings provide additional evidence for the 2nd rater.

Based on all of the evidence, independently select the overall Performance Level (Minimal, Basic, Proficient, or Advanced) that best characterizes the student’s current functioning.

Double check 1st rater’s total score for the students in Language Arts, Math, and Science to ensure all items were rated and the total scores are correct.

Compare their overall Performance Level determination to that of the 1st rater and decide if it is reliable using the Inter-Rater Reliability Estimate table of the MAAECF for Language Arts and Math. Settle any disagreements that result in unreliable decisions.

E & R Assessments 2007 31

Inter-Rater Agreement Table

E & R Assessments 2007 32

E & R Assessments 2007 33

Descriptive Statistics & Key Evidence to Support the Inferences Made From MAAECF Scores About

Student Achievement

E & R Assessments 2007 34

Descriptive Statistics for MAAECF Math

E & R Assessments 2007 35

Frequency Distributions for Math

E & R Assessments 2007 36

E & R Assessments 2007 37

Inter-rater Agreement & Evidence Quality Study

E & R Assessments 2007 38

Quality of Evidence Samples Matter!

E & R Assessments 2007 39

Factor Analytic Evidence to Support Claims about the MAAECF’s Structure

E & R Assessments 2007 40

Evidence Based on Relations to Other Variables

CAAVES 1%

Study

E & R Assessments 2007 41

More Evidence Based on Relations to Other Variables

CAAVES 1%

Study

E & R Assessments 2007 42

Evidence

based on

Consequences

of Testing

E & R Assessments 2007 43

Item and Scale Revision Plan in Progress for 2008

E & R Assessments 2007 44

MS CAARES Project Enhancements to the MAAECF

MS CAARES

is a new

GSEG Project

E & R Assessments 2007 45

Thank You Contact Information

Stephen N. Elliott Vanderbilt University 616-322-2538 steve.elliott@vanderbilt.edu

Andrew T. Roach Georgia State University

404-413-8176

aroach@gsu.edu