No Slide Title
Gary W. PhillipsAmerican Institutes for Research
CCSSO 2014 National Conference on Student Assessment (NCSA)New
OrleansJune 25-27, 2014
Multi State Alternate Assessment Collaborative: Measuring Growth
in Alternate Assessments
1Alternate Assessment State Collaborative
2Three State CollaborativesNational Center and State
Collaborative (NCSC) hosted by the National Center on Education
Outcomes (NCEO) at the University of Minnesota involving 24 states.
Dynamic Learning Maps (DLM) hosted by the Center for Educational
Testing and Evaluation (CETE) at the University of Kansas involving
13 states.Multistate Alternate Assessment Collaborative hosted by
the American Institutes for Research (AIR) in Washington, DC
involving 6 states.Reading, Writing (combine for ELA), Mathematics,
Science & Social Studies3Multistate Alternate Assessment
Collaborative
Multistate Alternate Assessment
CollaborativeState(firstYear)GradesCommonCoreStateStandardsPeerReviewApprovalVerticalScaleFT
Designs-independent-operational-embeddedGrowthPsychometricModelNM(2007)R,M
Gr. 3-4, 5-6, 7-8, 11-12S Gr. 3-4, 6-8, 11-12SS Gr. 11-12Extensions
to CCSS for R, MYesYesIFT & EFTAcross grades1 parameterPartial
CreditSC(2007)R,M Gr. 3-5, 6-8, HSS Gr. 3-5, 6-8Bio Gr. 10SS Gr.
3-5, 6-8Extensions to CCSS for R, MYesYesIFT & EFTAcross
grades
1 parameterPartial CreditDE(2011)R,M Gr. 2, 3-5, 6-8, 9-10S Gr.
5, 8, 10SS Gr. 4, 7, 9Extensions to CCSS for R, MTo
beSubmittedYesR,MIFT & EFTS,SSOFT & EFTAcross gradesfall to
spring1 parameterPartial CreditHI(2013)R,M Gr. 3-5, 6-8, 10S Gr. 4,
8, 10Extensions to CCSS for R, MTo beSubmittedYesOFT &
EFTAcross grades1 parameterPartial CreditOH(2013)R,M Gr. 3-5, 6-8,
HSS Gr. 5, 8, HSSS Gr. HSExtensions to CCSS for R, MTo
beSubmittedYesOFT & EFTAcross grades
1 parameterPartial CreditWY(2015)R, M, S Gr. 3-5, 6-8,
HSExtensions to CCSS for R, MTo beSubmittedYesOFT & EFTAcross
grades1 parameterPartial Credit4Multistate Alternate Assessment
Collaborative
Advantages of an Adaptive Alternate Assessment
5Advantages of Adaptive Alternate AssessmentStandardized
administration which allows the scores from the test to be
comparable from year-to-yearTest difficulty is adapted to student
abilityAdministered and scored by teachersHigh reliability of the
scores Aligned to Common Core State Standards (in most
states)Cheaper than portfolio assessmentsLess administration time
(about one hour per content area)
6Multistate Alternate Assessment Collaborative
\6Advantages of Adaptive Alternate AssessmentMeets the same
APA/AERA/NCME technical requirements as assessments of the general
populationGrowth modelsiPad pilot
7Multistate Alternate Assessment Collaborative
\7Administration
8AdministrationMultistate Alternate Assessment
Collaborative9Individually administered to students by the Test
AdministratorTakes about one hour per subjectA second rater
independently scores the test for a sample of students (used to
establish inter-rater reliability)
Calibration and Vertical Scale?
10Typical Linking Design & Vertical ScaleGrade BandUnique
TasksLinking TasksTotal for Operational TestTasks in Each
Grade-band Assessment Test Booklet1093126 863123 5912Total Number
of TasksAll Grades24630
11How do we Estimate Student Ability?
12Student Ability Estimation13Partial Credit Rasch ModelPattern
ScoringMultistate Alternate Assessment Collaborative
How are the Scores Reported?
14
Vertical Scale
19Vertical ScaleMultistate Alternate Assessment
Collaborative20Vertical ScaleMultistate Alternate Assessment
Collaborative21Growth
22Multistate Alternate Assessment Collaborative23
Multistate Alternate Assessment Collaborative24
Multistate Alternate Assessment Collaborative25
Multistate Alternate Assessment Collaborative26
How do we Establish Performance Standards?
27Standard Setting28Bookmark Procedure (or ID Matching)Workshop
panel of broadly representative stakeholdersOrdered Item Booklet
(OIB) covers Extended Common Core State Standards Response
Probability for ordering itemsTwo RoundsImpact dataAchievement
Level Descriptors (ALDs)Benchmarking (if requested by the
Department of Education)Vertical articulationMultiple Proficiency
levelsMultistate Alternate Assessment Collaborative
29Standard SettingOrdered Item Booklet .Most Difficult
Item22212019181716151413121110987654321Easiest ItemOrdered Item
Booklet
29EASIEST TO MOST DIFFICULTFOSTERS AN INTEGRATED CONCEPT OF HOW
THE TEST REFLECTS THE STANDARDSVEHICLE TO MAKE CUT SCORE
JUDGMENTSCOMMUNICATES HOW THE TRAIT INCREASES IN COMPLEXITY AS
ITEMS ASCEND THE SCALE30Standard SettingBookmark Procedure .Most
Difficult Item22212019181716151413121110987654321Easiest
ItemOrdered Item Booklet
30EASIEST TO MOST DIFFICULTFOSTERS AN INTEGRATED CONCEPT OF HOW
THE TEST REFLECTS THE STANDARDSVEHICLE TO MAKE CUT SCORE
JUDGMENTSCOMMUNICATES HOW THE TRAIT INCREASES IN COMPLEXITY AS
ITEMS ASCEND THE SCALE31Standard Setting I.D. Matching Procedure
.Most Difficult Item22212019181716151413121110987654321Easiest
ItemOrdered Item BookletConsistent withthe PLD for Meets
ProficiencyConsistent withthe PLD for
ApproachesProficiencyThreshold region
31EASIEST TO MOST DIFFICULTFOSTERS AN INTEGRATED CONCEPT OF HOW
THE TEST REFLECTS THE STANDARDSVEHICLE TO MAKE CUT SCORE
JUDGMENTSCOMMUNICATES HOW THE TRAIT INCREASES IN COMPLEXITY AS
ITEMS ASCEND THE SCALEStandard SettingArticulationMultistate
Alternate Assessment Collaborative32
Standard SettingArticulationMultistate Alternate Assessment
Collaborative33
Standard SettingImpact DataMultistate Alternate Assessment
Collaborative34Standard SettingImpact DataMultistate Alternate
Assessment Collaborative35OIB PagePercentageOIB PagePercentageOIB
PagePercentage199236045302962458462939425574727494265648275922755492569128545024788295251238863051522398431485323108332485422118133465521128034445620137835435719147736425817157537415916167338396015177139376113186940376213196841356312206642356492164433265722634431663Standard
SettingBenchmarking Against Previous StandardMultistate Alternate
Assessment Collaborative36OIB PagePercentageOIB PagePercentageOIB
PagePercentage19923604530296245846293942557472749426564827592275549256912854502478829525123886305152239843148532310833248542211813346552112803444562013783543571914773642581715753741591616733839601517*7139376113186940376213196841356312206642356492164433265722634431663*If
this page were selected during the standard setting it would
represent a level of proficiency comparable to the previous
Proficient Standard. Since we do not want to lower standards as we
transition to the new assessment, panelists should not set the
Proficient Standard lower than this page.Standard
SettingBenchmarking Against General Education StandardMultistate
Alternate Assessment Collaborative37OIB PagePercentageOIB
PagePercentageOIB
PagePercentage199236045302962458462939425574727494265648275922755492569128545024788295251238863051522398431485323108332485422118133465521128034445620137835435719147736*425817157537415916167338396015177139376113186940376213196841356312206642356492164433265722634431663*We
want the standard for the alternate assessment to be as challenging
for the alternate assessment population as the standard for the
general assessment is for the general population. The page
indicated above would represent such a standard for the alternate
assessment.
