View
1
Download
0
Category
Preview:
Citation preview
OregonDepartmentofEducation2016–2017TechnicalReportOregon’sAlternateAssessmentSystem
PeerReviewDocumentation:CriticalElements1-6
Oregon’sAlternateAssessmentSystemTechnicalReport:PeerReviewDocumentation:CriticalElements1-6
ItisthepolicyoftheStateBoardofEducationandapriorityoftheOregonDepartmentofEducationthattherewillbenodiscriminationorharassmentonthegroundsofrace,color,religion,sex,sexualorientation,nationalorigin,ageordisabilityinanyeducationalprograms,activitiesoremployment.PersonshavingquestionsaboutequalopportunityandnondiscriminationshouldcontacttheDeputySuperintendentofPublicInstructionwiththeOregonDepartmentofEducation. This technical report is one of a series that describes the development of Oregon’s Statewide Assessment System. The complete set of volumes provides comprehensive documentation of the development, procedures, technical adequacy, and results of the system.
2016-17ORExtendedAssessment PeerReviewCriticalElements–Page
3
TABLEOFCONTENTS
PEERREVIEWCRITICALELEMENTSREFERENCETABLES.....................................................................4OVERVIEW...........................................................................................................................................................14CRITICALELEMENT1:STATEWIDESYSTEMOFSTANDARDSANDASSESSMENTS....................161.1STATEADOPTIONOFACADEMICCONTENTSTANDARDSFORALLSTUDENTS....................................................161.2COHERENTANDRIGOROUSACADEMICCONTENTSTANDARDS..............................................................................221.3REQUIREDASSESSMENTS..............................................................................................................................................231.4POLICIESFORINCLUDINGALLSTUDENTSINASSESSMENTS..................................................................................231.5PARTICIPATIONDATA....................................................................................................................................................24
CRITICALELEMENT2–ASSESSMENTSYSTEMOPERATIONS............................................................262.1TESTDESIGNANDDEVELOPMENT..............................................................................................................................262.2ITEMDEVELOPMENT......................................................................................................................................................272.3TESTADMINISTRATION.................................................................................................................................................282.4MONITORINGTESTADMINISTRATION........................................................................................................................402.5TESTSECURITY................................................................................................................................................................422.6SYSTEMSFORPROTECTINGDATAINTEGRITYANDPRIVACY.................................................................................43
CRITICALELEMENT3–TECHNICALQUALITY:VALIDITY...................................................................463.1OVERALLVALIDITY,INCLUDINGVALIDITYBASEDONCONTENT..........................................................................463.2VALIDITYBASEDONCOGNITIVEPROCESSES.............................................................................................................513.3VALIDITYBASEDONINTERNALSTRUCTURE(CONTENTANDFUNCTION)..........................................................52ANNUALMEASUREABLEOBJECTIVESFREQUENCIES&PERCENTAGES.......................................................................643.4VALIDITYBASEDONRELATIONSTOOTHERVARIABLES........................................................................................72
CRITICALELEMENT4-TECHNICALQUALITY:OTHER.........................................................................774.1RELIABILITY.....................................................................................................................................................................77TESTINFORMATIONFUNCTIONS.........................................................................................................................................784.2FAIRNESSANDACCESSIBILITY......................................................................................................................................904.3FULLPERFORMANCECONTINUUM...........................................................................................................................1134.4SCORING.........................................................................................................................................................................1234.5MULTIPLEASSESSMENTFORMS................................................................................................................................1234.6MULTIPLEVERSIONSOFANASSESSMENT..............................................................................................................1234.7TECHNICALANALYSESANDONGOINGMAINTENANCE.........................................................................................124
CRITICALELEMENT5–INCLUSIONOFALLSTUDENTS.....................................................................1255.1PROCEDURESFORINCLUDINGSWDS......................................................................................................................1255.2A–5.2CPROCEDURESFORINCLUDINGELS...........................................................................................................1275.3ACCOMMODATIONS......................................................................................................................................................1275.4A–5.4EMONITORINGTESTADMINISTRATIONFORSPECIALPOPULATIONS..................................................128
CRITICALELEMENT6–ACADEMICACHIEVEMENTSTANDARDSANDREPORTING................1296.1STATEADOPTIONOFALTERNATEACADEMICACHIEVEMENTSTANDARDSFORSWSCD.............................1296.2ACHIEVEMENTSTANDARDSETTING........................................................................................................................1306.3CHALLENGINGANDALIGNEDACADEMICACHIEVEMENTSTANDARDS..............................................................1316.4REPORTING....................................................................................................................................................................133
CONCLUSIONSANDNEXTSTEPS................................................................................................................134APPENDIXTABLEANDDESCRIPTIONS...................................................................................................137
2016-17ORExtendedAssessment PeerReviewCriticalElements–Page
4
PeerReviewCriticalElementsReferenceTables
CriticalElementsCriticalElement1 StatewidesystemofstandardsandassessmentsCriticalElement2 AssessmentsystemoperationsCriticalElement3 Technicalquality-validityCriticalElement4 Technicalquality-otherCriticalElement5 InclusionofallstudentsCriticalElement6 Academicachievementstandardsandreporting
CriticalElements
CriticalElement1-Statewidesystemofstandardsandassessments1.1Stateadoptionofacademiccontentstandardsforallstudents
TheStateformallyadoptedchallengingacademiccontentstandardsforallstudentsinreading/languagearts,mathematicsandscienceandappliesitsacademiccontentstandardstoallpublicelementaryandsecondaryschoolsandstudentsintheState.
1.2Coherentandrigorousacademiccontentstandards
TheState’sacademiccontentstandardsinreading/languagearts,mathematicsandsciencespecifywhatstudentsareexpectedtoknowandbeabletodobythetimetheygraduatefromhighschooltosucceedincollegeandtheworkforce;containcontentthatiscoherent(e.g.,withinandacrossgrades)andrigorous;encouragetheteachingofadvancedskills;andweredevelopedwithbroadstakeholderinvolvement.
1.3RequiredAssessments
TheState’sassessmentsystemincludesannualgeneralandalternateassessments(basedongrade-levelacademicachievementstandardsoralternateacademicachievementstandards)in:Reading/languageartsandmathematicsineachofgrades3-8andatleastonceinhighschool(grades10-12);Scienceatleastonceineachofthreegradespans(3-5,6-9and10-12).
2016-17ORExtendedAssessment PeerReviewCriticalElements–Page
5
1.4Policiesforincludingallstudentsinassessments
TheStaterequirestheinclusionofallpublicelementaryandsecondaryschoolstudentsinitsassessmentsystemandclearlyandconsistentlycommunicatesthisrequirementtodistrictsandschools.Forstudentswithdisabilities,policiesstatethatallstudentswithdisabilitiesintheState,includingstudentswithdisabilitiespubliclyplacedinprivateschoolsasameansofprovidingspecialeducationandrelatedservices,mustbeincludedintheassessmentsystem;ForEnglishLearners:
A)PoliciesstatethatallEnglishlearnersmustbeincludedintheassessmentsystem,unlesstheStateexemptsastudentwhohasattendedschoolsintheU.S.forlessthan12monthsfromoneadministrationofitsreading/languageartsassessment;
B) IftheStateadministersnativelanguageassessments,theStaterequiresEnglishlearnerstobeassessedinreading/languageartsinEnglishiftheyhavebeenenrolledinU.S.schoolsforthreeormoreconsecutiveyears,exceptifadistrictdetermines,onacase-by-casebasis,thatnativelanguageassessmentswouldyieldmoreaccurateandreliableinformation,thedistrictmayassessastudentwithnativelanguageassessmentsforaperiodnottoexceedtwoadditionalconsecutiveyears.
1.5ParticipationData
TheState’sparticipationdatashowthatallstudents,disaggregatedbystudentgroupandassessmenttype,areincludedintheState’sassessmentsystem.Inaddition,iftheStateadministersend-of-courseassessmentsforhighschoolstudents,theStatehasproceduresinplaceforensuringthateachstudentistestedandcountedinthecalculationofparticipationratesoneachrequiredassessmentandprovidesthecorrespondingdata.
2016-17ORExtendedAssessment PeerReviewCriticalElements–Page
6
CriticalElement2-Assessmentsystemoperations2.1TestDesignandDevelopment
TheState’stestdesignandtestdevelopmentprocessiswell-suitedforthecontent,istechnicallysound,alignstheassessmentstothefullrangeoftheState’sacademiccontentstandards,andincludes:
A) Statement(s)ofthepurposesoftheassessmentsandtheintendedinterpretationsandusesofresults;
B) Testblueprintsthatdescribethestructureofeachassessmentinsufficientdetailtosupportthedevelopmentofassessmentsthataretechnicallysound,measurethefullrangeoftheState’sgrade-levelacademiccontentstandards,andsupporttheintendedinterpretationsandusesoftheresults;
C) ProcessestoensurethateachassessmentistailoredtotheknowledgeandskillsincludedintheState’sacademiccontentstandards,reflectsappropriateinclusionofchallengingcontent,andrequirescomplexdemonstrationsorapplicationsofknowledgeandskills(i.e.,higher-orderthinkingskills);
D) IftheStateadministerscomputer-adaptiveassessments,theitempoolanditemselectionproceduresadequatelysupportthetestdesign.
2.2ItemDevelopment
TheStateusesreasonableandtechnicallysoundprocedurestodevelopandselectitemstoassessstudentachievementbasedontheState’sacademiccontentstandardsintermsofcontentandcognitiveprocess,includinghigher-orderthinkingskills.
2.3TestAdministration
TheStateimplementspoliciesandproceduresforstandardizedtestadministration,specificallytheState:
A) Hasestablishedandcommunicatestoeducatorsclear,thoroughandconsistentstandardizedproceduresfortheadministrationofitsassessments,includingadministrationwithaccommodations;
B) HasestablishedprocedurestoensurethatallindividualsresponsibleforadministeringtheState’sgeneralandalternateassessmentsreceivetrainingontheState’sestablishedproceduresfortheadministrationofitsassessments;
C) IftheStateadministerstechnology-basedassessments,theStatehasdefinedtechnologyandotherrelatedrequirements,includedtechnology-basedtestadministrationinitsstandardizedproceduresfortestadministration,andestablishedcontingencyplanstoaddresspossibletechnologychallengesduringtestadministration.
2.4Monitoringtestadministration
TheStateadequatelymonitorstheadministrationofitsStateassessmentstoensurethatstandardizedtestadministrationproceduresareimplementedwithfidelityacrossdistrictsandschools.
2016-17ORExtendedAssessment PeerReviewCriticalElements–Page
7
2.5TestSecurity TheStatehasimplementedanddocumentedanappropriatesetofpoliciesandprocedurestopreventtestirregularitiesandensuretheintegrityoftestresultsthrough:
A) Preventionofanyassessmentirregularities,includingmaintainingthesecurityoftestmaterials,propertestpreparationguidelinesandadministrationprocedures,incident-reportingprocedures,consequencesforconfirmedviolationsoftestsecurity,andrequirementsforannualtrainingatthedistrictandschoollevelsforallindividualsinvolvedintestadministration;
B) Detectionoftestirregularities;C) Remediationfollowinganytestsecurityincidentsinvolvingany
oftheState’sassessments;D) Investigationofallegedorfactualtestirregularities.
2.6Systemsforprotectingdataintegrityandprivacy
TheStatehaspoliciesandproceduresinplacetoprotecttheintegrityandconfidentialityofitstestmaterials,test-relateddata,andpersonallyidentifiableinformation,specifically:
A) Toprotecttheintegrityofitstestmaterialsandrelateddataintestdevelopment,administration,andstorageanduseofresults;
B) Tosecurestudent-levelassessmentdataandprotectstudentprivacyandconfidentiality,includingguidelinesfordistrictsandschools;
C) Toprotectpersonallyidentifiableinformationaboutanyindividualstudentinreporting,includingdefiningtheminimumnumberofstudentsnecessarytoallowreportingofscoresforallstudentsandstudentgroups.
CriticalElement3-Technicalquality-validity3.1Overallvalidity,includingvaliditybasedoncontent
TheStatehasdocumentedadequateoverallvalidityevidenceforitsassessments,andtheState’svalidityevidenceincludesevidencethattheState’sassessmentsmeasuretheknowledgeandskillsspecifiedintheState’sacademiccontentstandards,including:
A) DocumentationofadequatealignmentbetweentheState’sassessmentsandtheacademiccontentstandardstheassessmentsaredesignedtomeasureintermsofcontent(i.e.,knowledgeandprocess),thefullrangeoftheState’sacademiccontentstandards,balanceofcontent,andcognitivecomplexity;
B) IftheStateadministersalternateassessmentsbasedonalternateacademicachievementstandards,theassessmentsshowadequatelinkagetotheState’sacademiccontentstandardsintermsofcontentmatch(i.e.,nounrelatedcontent)andthebreadthofcontentandcognitivecomplexitydeterminedintestdesigntobeappropriateforstudentswiththemostsignificantcognitivedisabilities.
2016-17ORExtendedAssessment PeerReviewCriticalElements–Page
8
3.2Validitybasedoncognitiveprocesses
TheStatehasdocumentedadequatevalidityevidencethatitsassessmentstaptheintendedcognitiveprocessesappropriateforeachgradelevelasrepresentedintheState’sacademiccontentstandards.
3.3Validitybasedoninternalstructure
TheStatehasdocumentedadequatevalidityevidencethatthescoringandreportingstructuresofitsassessmentsareconsistentwiththesub-domainstructuresoftheState’sacademiccontentstandardsonwhichtheintendedinterpretationsandusesofresultsarebased.
3.4Validitybasedonrelationstoothervariables
TheStatehasdocumentedadequatevalidityevidencethattheState’sassessmentscoresarerelatedasexpectedwithothervariables.
CriticalElement4-Technicalquality-other4.1Reliability TheStatehasdocumentedadequatereliabilityevidenceforits
assessmentsforthefollowingmeasuresofreliabilityfortheState'sstudentpopulationoverallandeachstudentgroupand,iftheState'sassessmentsareimplementedinmultipleStates,fortheassessmentoverallandeachstudentgroup,including:
A) TestreliabilityoftheState'sassessmentsestimatedforitsstudentpopulation;
B) OverallandconditionalstandarderrorofmeasurementoftheState'sassessments;
C) Consistencyandaccuracyofestimatesincategoricalclassificationdecisionsforthecutscoresandachievementlevelsbasedontheassessmentresults;
Forcomputer-adaptivetests,evidencethattheassessmentsproducetestformswithadequatelypreciseestimatesofastudent'sachievement.
4.2Fairnessandaccessibility
TheStatehastakenreasonableandappropriatestepstoensurethatitsassessmentsareaccessibletoallstudentsandfairacrossstudentgroupsinthedesign,developmentandanalysisofitsassessments.
4.3Fullperformancecontinuum
TheStatehasensuredthateachassessmentprovidesanadequatelypreciseestimateofstudentperformanceacrossthefullperformancecontinuum,includingforhigh-andlow-achievingstudents.
4.4Scoring TheStatehasestablishedanddocumentedstandardizedscoringproceduresandprotocolsforitsassessmentsthataredesignedtoproducereliableresults,facilitatevalidscoreinterpretations,andreportassessmentresultsintermsoftheState’sacademicachievementstandards.
2016-17ORExtendedAssessment PeerReviewCriticalElements–Page
9
4.5Multipleassessmentforms
IftheStateadministersmultipleformswithinacontentareaandgradelevel,withinoracrossschoolyears,theStateensuresthatallformsadequatelyrepresenttheState’sacademiccontentstandardsandyieldconsistentscoreinterpretationssuchthattheformsarecomparablewithinandacrossschoolyears.
4.6Multipleversionsofanassessment
IftheStateadministersassessmentsinmultipleversionswithinacontentarea,gradelevel,orschoolyear,theState:
A) Followedadesignanddevelopmentprocesstosupportcomparableinterpretationsofresultsforstudentstestedacrosstheversionsoftheassessments;
B) Documentedadequateevidenceofcomparabilityofthemeaningandinterpretationsoftheassessmentresults.
4.7Technicalanalysesandongoingmaintenance
TheStatehasasystemformonitoringandmaintaining,andimprovingasneeded,thequalityofitsassessmentsystem,includingclearandtechnicallysoundcriteriafortheanalysesofalloftheassessmentsinitsassessmentsystem(i.e.,generalassessmentsandalternateassessments).
2016-17ORExtendedAssessment PeerReviewCriticalElements–Page
10
CriticalElement5-Inclusionofallstudents5.1ProceduresforincludingSWDs
TheStatehasinplaceprocedurestoensuretheinclusionofallpublicelementaryandsecondaryschoolstudentswithdisabilitiesintheState’sassessmentsystem,including,ataminimum,guidanceforIEPTeamstoinformdecisionsaboutstudentassessmentsthat:
A) Providesclearexplanationsofthedifferencesbetweenassessmentsbasedongrade-levelacademicachievementstandardsandassessmentsbasedonalternateacademicachievementstandards,includinganyeffectsofStateandlocalpoliciesonastudent’seducationresultingfromtakinganalternateassessmentbasedonalternateacademicachievementstandards;
B) Statesthatdecisionsabouthowtoassessstudentswithdisabilitiesmustbemadebyastudent’sIEPTeambasedoneachstudent’sindividualneeds;
C) Providesguidelinesfordeterminingwhethertoassessastudentonthegeneralassessmentwithoutaccommodation(s),thegeneralassessmentwithaccommodation(s),oranalternateassessment;
D) Providesinformationonaccessibilitytoolsandfeaturesavailabletostudentsingeneralandassessmentaccommodationsavailableforstudentswithdisabilities;
E) Providesguidanceregardingselectionofappropriateaccommodationsforstudentswithdisabilities;
F) IncludesinstructionsthatstudentseligibletobeassessedbasedonalternateacademicachievementstandardsmaybefromanyofthedisabilitycategorieslistedintheIDEA;
G) Ensuresthatparentsofstudentswiththemostsignificantcognitivedisabilitiesareinformedthattheirstudent’sachievementwillbebasedonalternateacademicachievementstandardsandofanypossibleconsequencesoftakingthealternateassessmentsresultingfromdistrictorStatepolicy(e.g.,ineligibilityforaregularhighschooldiplomaifthestudentdoesnotdemonstrateproficiencyinthecontentareaontheState’sgeneralassessments);
H) TheStatehasproceduresinplacetoensurethatitsimplementationofalternateacademicachievementstandardsforstudentswiththemostsignificantcognitivedisabilitiespromotesstudentaccesstothegeneralcurriculum.
2016-17ORExtendedAssessment PeerReviewCriticalElements–Page
11
5.2ProceduresforincludingELs
TheStatehasinplaceprocedurestoensuretheinclusionofallEnglishlearnersinpublicelementaryandsecondaryschoolsintheState’sassessmentsystemandclearlycommunicatesthisinformationtodistricts,schools,teachers,andparents,including,ataminimum:
A) ProceduresfordeterminingwhetheranEnglishlearnershouldbeassessedwithaccommodation(s);
B) InformationonaccessibilitytoolsandfeaturesavailabletoallstudentsandassessmentaccommodationsavailableforEnglishlearners;
C) GuidanceregardingselectionofappropriateaccommodationsforEnglishlearners.
5.3Accommodations
TheStatemakesavailableappropriateaccommodationsandensuresthatitsassessmentsareaccessibletostudentswithdisabilitiesandEnglishlearners.Specifically,theState:
A) EnsuresthatappropriateaccommodationsareavailableforstudentswithdisabilitiesunderIDEAandstudentscoveredbySection504;
B) EnsuresthatappropriateaccommodationsareavailableforEnglishlearners;
C) Hasdeterminedthattheaccommodationsitprovides(i)areappropriateandeffectiveformeetingtheindividualstudent’sneed(s)toparticipateintheassessments,(ii)donotaltertheconstructbeingassessed,and(iii)allowmeaningfulinterpretationsofresultsandcomparisonofscoresforstudentswhoneedandreceiveaccommodationsandstudentswhodonotneedanddonotreceiveaccommodations;
D) Hasaprocesstoindividuallyreviewandallowexceptionalrequestsforasmallnumberofstudentswhorequireaccommodationsbeyondthoseroutinelyallowed.
5.4Monitoringtestadministrationforspecialpopulations
TheStatemonitorstestadministrationinitsdistrictsandschoolstoensurethatappropriateassessments,withorwithoutappropriateaccommodations,areselectedforstudentswithdisabilitiesunderIDEA,studentscoveredbySection504,andEnglishlearnerssothattheyareappropriatelyincludedinassessmentsandreceiveaccommodationsthatare:
A) ConsistentwiththeState’spoliciesforaccommodations;B) Appropriateforaddressingastudent’sdisabilityorlanguage
needsforeachassessmentadministered;C) Consistentwithaccommodationsprovidedtothestudents
duringinstructionand/orpractice;D) Consistentwiththeassessmentaccommodationsidentifiedbya
student’sIEPTeamor504teamforstudentswithdisabilities,oranotherprocessforanEnglishlearner;
E) Administeredwithfidelitytotestadministrationprocedures.
2016-17ORExtendedAssessment PeerReviewCriticalElements–Page
12
CriticalElement6-Academicachievementstandardsandreporting6.1Stateadoptionofacademicachievementstandardsforallstudents
TheStateformallyadoptedchallengingacademicachievementstandardsinreading/languagearts,mathematicsandinscienceforallstudents,specifically:
A) TheStateformallyadoptedacademicachievementstandardsintherequiredtestedgradesand,atitsoption,alsoalternateacademicachievementstandardsforstudentswiththemostsignificantcognitivedisabilities;
B) TheStateappliesitsgrade-levelacademicachievementstandardstoallpublicelementaryandsecondaryschoolstudentsenrolledinthegradetowhichtheyapply,withtheexceptionofstudentswiththemostsignificantcognitivedisabilitiestowhomalternateacademicachievementstandardsmayapply;
C) TheState’sacademicachievementstandardsand,asapplicable,alternateacademicachievementstandards,include:(a)Atleastthreelevelsofachievement,withtwoforhighachievementandathirdforlowerachievement;(b)descriptionsofthecompetenciesassociatedwitheachachievementlevel;and(c)achievementscoresthatdifferentiateamongtheachievementlevels.
6.2Achievementstandardsetting
TheStateusedatechnicallysoundmethodandprocessthatinvolvedpanelistswithappropriateexperienceandexpertiseforsettingitsacademicachievementstandardsandalternateacademicachievementstandardstoensuretheyarevalidandreliable.
6.3Challengingandalignedacademicachievementstandards
TheState’sacademicachievementstandardsarechallengingandalignedwiththeState’sacademiccontentstandardssuchthatahighschoolstudentwhoscoresattheproficientorabovelevelhasmasteredwhatstudentsareexpectedtoknowandbeabletodobythetimetheygraduatefromhighschoolinordertosucceedincollegeandtheworkforce.IftheStatehasdefinedalternateacademicachievementstandardsforstudentswiththemostsignificantcognitivedisabilities,thealternateacademicachievementstandardsarelinkedtotheState’sgrade-levelacademiccontentstandardsorextendedacademiccontentstandards,showlinkagetodifferentcontentacrossgrades,andreflectprofessionaljudgmentofthehighestachievementstandardspossibleforstudentswiththemostsignificantcognitivedisabilities.
2016-17ORExtendedAssessment PeerReviewCriticalElements–Page
13
6.4Reporting TheStatereportsitsassessmentresults,andthereportingfacilitatestimely,appropriate,credible,anddefensibleinterpretationsandusesofresultsforstudentstestedbyparents,educators,Stateofficials,policymakersandotherstakeholders,andthepublic,including:
A) TheStatereportstothepublicitsassessmentresultsonstudentachievementateachproficiencylevelandthepercentageofstudentsnottestedforallstudentsandeachstudentgroupaftereachtestadministration;
B) TheStatereportsassessmentresults,includingitemizedscoreanalyses,todistrictsandschoolssothatparents,teachers,principals,andadministratorscaninterprettheresultsandaddressthespecificacademicneedsofstudents,andtheStatealsoprovidesinterpretiveguidestosupportappropriateusesoftheassessmentresults;
C) TheStateprovidesfortheproductionanddeliveryofindividualstudentinterpretive,descriptive,anddiagnosticreportsaftereachadministrationofitsassessmentsthat:1) Providevalidandreliableinformationregardinga
student’sachievement;2) Reportthestudent’sachievementintermsoftheState’s
grade-levelacademicachievementstandards(includingperformance-leveldescriptors);
3) Provideinformationtohelpparents,teachers,andprincipalsinterpretthetestresultsandaddressthespecificacademicneedsofstudents;
4) Areavailableinalternateformats(e.g.,Brailleorlargeprint)uponrequestand,totheextentpracticable,inanativelanguagethatparentscanunderstand;
5) TheStatefollowsaprocessandtimelinefordeliveringindividualstudentreportstoparents,teachers,andprincipalsassoonaspracticableaftereachtestadministration.
2016-2017ORExtendedAssessment Overview&CriticalElement1–Page14
OverviewThisdocumentprovidesupdatedtechnicaladequacydocumentationfortheOregonExtendedAssessment(ORExt),whichisOregon'salternateassessmentbasedonalternateacademicachievementstandards(AA-AAAS).Thedocumentationincludestestdesignanddevelopment,technicalcharacteristicsoftheassessments,andtheiruses,andimpactinprovidingproficiencydataongradelevelstatestandardsaspartofthemandatesfromtheEveryStudentSucceedsActof2015(ESSA).TheORExtassessmentswereredesignedin2014-15,includingaverticalscaleinGrades3-8inEnglishlanguageartsandmathematicstosupporteventualdeterminationsofstudentgrowthovertime.ThetestisalignedtoEssentializedStandards(EsSt)thatarepartofcomprehensiveEssentializedAssessmentFrameworks(EAFs)thatwerewrittenatthreelevelsofcomplexity(low,medium,andhigh).TheEsSthavebeenlinkedtogradelevelcontentandexpectations,butsystematicallyreducedintermsofdepth,breadth,andcomplexity(RDBC).AllORExtitemsemployedinthe2016-17ORExtadministrationweredevelopedin2014-15.AnalignmentstudywasconductedatthattimeanditwasdeterminedthatallitemswerealignedtothenewEsSt.AstatewidesampleofOregongeneralandspecialeducationteachershavereviewedalltestitemsfor:1)alignmenttotheEAFs,2)accessibilityforstudentswithsignificantcognitivedisabilities,3)sensitivity,and4)bias.Alloperationalitemsmettheestablishedcriteria.Inaddition,AchievementLevelDescriptors(ALDs)werealsoreviewedforalignmenttotheEsSt.SeeSections1.1,1.2,6.1,and6.3foradditionalinformationrelatedtothecomprehensivegradelevelstandardstoEsStlinkage,aswellasalignmentofitemstotheEsSt.TheORExttestdesignsupportsstudentaccess,includingaccesstoreadaloudfordirectionsandprompts,presentationofoneitemperpage,anditemsdesignedatthreelevelsofcomplexitywherethelowlevelcomplexityitemsincludegraphicand/orobjectsupport.Forassessors,thescoringprocesshasalsobeensimplified,withanswersbeingeithercorrect(1)orincorrect(0).PartialcreditisnolongerpartofthescoringmetricfortheORExt.Inaddition,theoneitemperpageformatnotonlyincreasesstudentabilitytofocusattention,butalsoreducestheburdenonassessorstomaskitemsthatarenotbeingtested.Thefieldappearstohavebeenappreciativeoftheredesign,particularlytheEssentializedStandardsandnewaccessandefficiencyfeatures.Inadditiontodevelopingandreviewing/editingover5,000newitems,conductinganoperationalfieldtest,anddevelopingaverticalscale,thedevelopmentofanewORExtrequiredthatnewAlternateAcademicAchievementStandards(AAAAS)bedevelopedandapproved.ComprehensiveStandardSettingmeetingswereconductedonJune15-17,2015,whichwerethenapprovedbytheOregonStateBoardofEducationonJune25,2015,includingnewachievementleveldescriptors(ALDs)andcutscoresfortheassessments.ComprehensiveAnnualMeasureableObjective(AMO)reportswerefinalizedonJuly10,2015.
2016-2017ORExtendedAssessment Overview&CriticalElement1–Page15
Thoughanalignmentstudywasconductedinthefallof2014asdescribedabove,Non-RegulatoryGuidancefromtheU.S.DepartmentofEducation,publishedonSeptember25,2015,includedanexpectationthatallalignmentstudiesmustbeindependent(seeCriticalElement3.1).Anindependentcontractor,Dr.DiannaCarrizales,wasthereforehiredtoperformanadditionalalignmentstudyinthespringof2017.Completeresultsarepresentedinthistechnicalreport(seeSection3.1A-B&3.2).Inadditiontotheindependentalignmentstudy,apilottabletadministrationstudywasalsoconductedinthe2016-17schoolyear.Thisisthesecondphaseofathree-yearplantomaketabletadministrationoftheORExtavailablebythe2017-18schoolyear.AsummaryofthepilottabletadministrationstudyisprovidedinSection2.3C.Completeresultsfromthepilottabletadministrationstudy,phasetwo,arepresentedinAppendix2.3C.Theindependentalignmentstudyandpilottabletadministrationstudyarepartofourfive-yeartechnicaldocumentationplan.Futureplansincludeaninter-raterreliabilitystudy,aswellasanalysesoftheimpactofaccommodations.
2016-2017ORExtendedAssessment Overview&CriticalElement1–Page16
CriticalElement1:StatewideSystemofStandardsandAssessments1.1StateAdoptionofAcademicContentStandardsforAllStudentsTheOregonStateBoardofEducation(SBE)adoptednew,challengingacademiccontentstandards,theCommonCoreStateStandards(CCSS),inEnglishlanguageartsandmathematicsinGradesK-12onOctober28,2010.TheseCCSSareutilizedforallstudentsinOregon'spublicschools.OregonwasactivelyinvolvedinthedevelopmentoftheCCSS,astheOregonDepartmentofEducation(ODE),theEducationalEnterpriseSteeringCommittee(EESC),Oregon'sEducationServiceDistricts,andschooldistrictrepresentativesprovidedfeedbackonthedraftCCSSstandards.Similarly,theSBEadoptedtheNextGenerationScienceStandards(NGSS)onMarch6,2014.TheNGSSestablishlearningtargetsforallstudentsinOregon'spublicschoolsinGradesK-12.TheODEandtheOregonScienceContentandAssessmentPanelprovideddirectfeedbackrelatedtotheNGSS.TheNGSSarebeingphasedinovertimeinstructionally,sostudentsarebeingassessedrelativetotheOregonScience(ORSci)standardsthatwereadoptedin2009.Forstatecontentstandards,seehttp://www.oregon.gov/ode/educator-resources/standards/Pages/default.aspx.Thetablebelowprovidesexamplesofessentializedstandardsingrades5,8,&11inthesubjectareasofEnglishlanguagearts(ELA),mathematics,andscience.CompleteEAFspreadsheetsareavailableatthelinkprovidedhere,aswell.Intherightcolumnaredesignationsforestimateddifficultyofanitem:L(low),M(medium),andH(high).
http://www.brtprojects.org/publications/training-modulesSeeAppendix1.1foraUserGuidethatexplainsthedevelopmentprocessandintendedusesfortheEAFs.
2016-2017ORExtendedAssessment Overview&CriticalElement1–Page17
GRADE5
Area Cluster Standard Sub-StandardEssential-izedStandard
L/M/HDescriptors
ReadingStandardsforLiteratureK–5
1.KeyIdeasandDetails
3.Compareandcontrast2ormorecharacters,settings,oreventsinastoryordrama,drawingonspecificdetailsinthetext(e.g.,howcharactersinteract).
None
Identifyacharacter,setting,oreventinastoryreadtostudent.
L-Sentenceof7wordsorlessthatcontains1character,setting,oreventreadtostudent.M-2shortsentencesthatcontain1character,setting,oreventreadtostudent.H-2mediumsentencesthatcontain1character,setting,oreventreadtostudent.
MathNumber&OperationsinBaseTen
1.Understandtheplacevaluesystem.
1.Recognizethatinamulti-digitnumber,adigitinoneplacerepresents10timesasmuchasitrepresentsintheplacetoitsrightand1/10ofwhatitrepresentsintheplacetoitsleft.
Useplacevaluetocomparenumbersthataremultiplesof10andones'versustens'placeand.5.
L-identifymultiplesof10:10,20,30,40,50,60.M-identifytherelationbetweentheplacevaluesforthedouble-digitnumbers11,22,33,44,55.H-identifywhichnumberisintheten'splaceandone'splace.
2016-2017ORExtendedAssessment Overview&CriticalElement1–Page18
Area Cluster Standard Sub-StandardEssential-izedStandard
L/M/HDescriptors
Science*MatterandItsInteractions
NGSSStandard:Measureandgraphquantitiestoprovideevidencethatregardlessofthetypeofchangethatoccurswhenheating,cooling,ormixingsubstances,thetotalweightofmatterisconserved.
ORScienceStandards:5.3S.1Basedonobservationsandscienceprinciples,identifyquestionsthatcanbetested,designanexperimentorinvestigation,andidentifyappropriatetools.Collectandrecordmultipleobservationswhileconductinginvestigationsorexperimentstotestascientificquestionorhypothesis.5.3S.2Identifypatternsindatathatsupportareasonableexplanationfortheresultsofaninvestigationorexperimentandcommunicatefindingsusinggraphs,charts,maps,models,andoralandwrittenreports.
Measureand/orcomparetheweightofdifferenttypesofmatter.
L-Measuretheweight/massofcommonobjectsinvariousphasesofmatterusingpicturesofsuchobjects(i.e.,anobjectonascalethatweighs3pounds);M-Comparetheweight/massofcommonobjectsinvariousstagesofmatterusingpicturesofsuchobjects(e.g.,aballoonweighslessthanarockorglassofwater)-Choosethecorrecttooltomeasuretheweight/massofobjects;H-Comparetheweight/massofcommonobjectsinvariousphasesofmatterusinggraphsanddata.
Note.Thescienceessentializedstandardsaredually-linkedtobothNGSSandOregonSciencestandards,respectively.BothgeneraleducationstandardsarethuslistedforscienceintheseEAFtables.
2016-2017ORExtendedAssessment Overview&CriticalElement1–Page19
GRADE8Area Cluster Standard Sub-Standard Essentialized
Standard L/M/HDescriptors
ReadingStandardsforLiterature6–12
2.CraftandStructure
6.Analyzehowdifferencesinthepointsofviewofthecharactersandtheaudienceorreader(e.g.,createdthroughtheuseofdramaticirony)createsucheffectsassuspenseorhumor.
None
Identifythenarratororacharacterinastoryreadtostudent.
L-3sentencesthatcontain2charactersornarratorsreadtostudent.M-Paragraphof4sentencesthatcontains2charactersornarratorsreadtostudent.H-Paragraphof5sentencesthatcontains2charactersornarratorsreadtostudent.
MathStatistics&Probability
1.Investigatepatternsofassociationinbivariatedata.
3.Usetheequationofalinearmodeltosolveproblemsinthecontextofbivariatemeasurementdata,interpretingtheslopeandintercept.Forexample,inalinearmodelforabiologyexperiment,interpretaslopeof1.5cm/hrasmeaningthatanadditionalhourofsunlighteachdayisassociatedwithanadditional1.5cminmatureplantheight.
Compareratesusingslower/less,faster/more,same(mph,beatspersecond,$perhour,$perlb).
L-identifyfasterrateusing(0-20).M-identifyslower,faster,orsamerateusing(21-50).H-identifyslower,faster,orsamerateusing(51-100).
Science Energy
NGSSStandard:Plananinvestigationtodeterminetherelationshipsamongtheenergytransferred,thetypeofmatter,themass,andthechangeintheaveragekineticenergyoftheparticlesasmeasuredbythetemperatureofthesample.
ORScienceStandards8.2P.2Explainhowenergyistransferred,transformed,andconserved.
Recognizetemperatureasameasureofhowhotorcoldmatteris,andthatheatistransferable.
L-Recognizethedifferencebetweenhotandcold(e.g.,objects,outside);M-Recognizethathotandcoldarerelatedtomeasuresoftemperature,includingchangesintemperature;H-Identifyexamplesofheattransfer,andhowsuchtransfermightbeminimized/maximized(e.g.,wearingacoattostaywarm).
2016-2017ORExtendedAssessment Overview&CriticalElement1–Page20
GRADE11Area Cluster Standard Sub-
StandardEssentializedStandard L/M/HDescriptors
ReadingStandardsforLiterature6–12
2.CraftandStructure
4.Determinethemeaningofwordsandphrasesastheyareusedinthetext,includingfigurativeandconnotativemeanings;analyzetheimpactofspecificwordchoicesonmeaningandtone,includingwordswithmultiplemeaningsorlanguagethatisparticularlyfresh,engaging,orbeautiful.(IncludeShakespeareaswellasotherauthors.)
None
Identifythemeaningoffigurative,connotative,orwordswith2ormoremeanings.
L-Paragraphof4sentencesreadtostudent.M-Paragraphof5sentencesreadtostudent.H-2paragraphsreadtostudent.
Math
ExpressingGeometricPropertieswithEquations
2.Usecoordinatestoprovesimplegeometrictheoremsalgebraically
7.Usecoordinatestocomputeperimetersofpolygonsandareasoftrianglesandrectangles,e.g.,usingthedistanceformula.
Identifytheperimeteroftriangles,squares,rectangles,andpentagons.
L-identifyperimeteroftriangleswithsidelengths(1-5).M-identifytheperimeterofsquaresandrectangleswithsidelengths(1-10).H-identifytheperimeterofpentagonswithsidelengths(1-20).
2016-2017ORExtendedAssessment Overview&CriticalElement1–Page21
Area Cluster Standard Sub-Standard
EssentializedStandard L/M/HDescriptors
Science Earth’sSystems
DevelopamodeltoillustratehowEarth’sinternalandsurfaceprocessesoperateatdifferentspatialandtemporalscalestoformcontinentalandocean-floorfeatures.
H.1E.2DescribethestructureandcompositionofEarth’satmosphere,geosphere,andhydrosphere.H.2E.1Identifyandpredicttheeffectofenergysources,physicalforces,andtransferprocessesthatoccurintheEarthsystem.Describehowmatterandenergyarecycledbetweensystemcomponentsovertime.H.2E.2ExplainhowEarth’satmosphere,geosphere,andhydrospherechangeovertimeandatvaryingrates.ExplaintechniquesusedtoelucidatethehistoryofeventsonEarth.
Identifydifferent(geoscience)processesthatshapetheEarthincludingassociatedEarthfeatures.(S08ESS2.2)
L-Identifyconditionsthatleadtospecifictypesofsurfaceweathering(i.e.,withwater,ice,orwindasvehicle-Whichshowswatererosion?-ariver,pondorvolcano);M-Identifygeoscienceprocessesthatshapelocalgeographicfeatures(e.g.,earthquakes,volcanoes,meteorites/craters-Whichisanexampleofvolcanism?–picturesofavolcano,river,rain);H-ExtendM-levelquestionsbylinkingfeaturestothegeoscienceprocess(e.g.,Whichtypeoferosionprocesslikelyledtothecanyon?-river,rain,wind;Whichfeatureisassociatedwithrecentvolcanism?-island,valley,river).
2016-2017ORExtendedAssessment Overview&CriticalElement1–Page22
1.2CoherentandrigorousAcademicContentStandardsTheCCSS,ORSci,andNGSSdefinewhatstudentsinOregonshouldknowandbeabletodobythetimetheygraduatefromhighschool.TheseCCSS,whichweredevelopedbynationalstakeholdersandeducationexperts,havebeendeterminedtobecoherentandrigorousbyresearchersattheFordhamInstitute(seeAppendix1.2).Theywerealsodevelopedwithwidestakeholderinvolvement,particularlyhereinOregon.ThenewORExtislinkeddirectlytothecontentintheCCSSinEnglishlanguagearts(reading,writing,&language)andmathematics.TheORExtisduallylinkedtotheORSciaswellastheNGSS.TheNGSSarewidelyacceptedbymostrelevantscienceinstructionorganizationsasreflectiveofrigorousandcoherentscienceconcepts(seehttp://www.nextgenscience.org/support-scientific-engineering-and-education-communities).ThenewEssentializedAssessmentFrameworks(EAFs)arepubliclyavailableatthelinkonpage16aboveintheEssentializedAssessmentFrameworkssection.AUserGuideisprovidedtoinstructeducatorsregardingtheintendedusesoftheEssentializedStandards(EsSt),includingthedevelopmentofPresentLevelsofAcademicAchievementandFunctionalPerformance(PLAAFP)andIndividualizedEducationProgram(IEP)goalsandobjectives.ThebasicessentializationprocessemployedtogenerateessentializedstandardsandwritealigneditemsfortheORExtisoutlinedbelow.Theprocesscanalsobeusedtosupportthedevelopmentofcurricularandinstructionalmaterials,foundedinresearch-basedpedagogy.
2016-2017ORExtendedAssessment Overview&CriticalElement1–Page23
1.3RequiredAssessmentsTheORExtassessmentswereadministeredinthe2016-17schoolyearinELAandmathinGrades3-8andonceinGrade11;scienceisassessedinGrades5,8,&11.ThisassessmentplanmeetstherequirementsforgradelevelassessmentinGrades3-8andonceinhighschool(Grades10-12)forELAandmathematics,whilescienceisassessedonceinthe3-5gradeband,onceinthe6-9gradeband,andonceinthe10-12gradeband:ContentArea Grade
3Grade4
Grade5
Grade6
Grade7
Grade8
Grade11
Englishlanguagearts X X X X X X XMathematics X X X X X X XScience X X X1.4PoliciesforIncludingAllStudentsinAssessmentsOriginally,Oregonstatuterequiredthatallstudentsparticipateinstatewideassessments,withexceptionsallowedfordistrict-approvedparentrequestforassessmentwaivers(parentopt-outrequests)relatedtostudentdisabilityorreligiousbeliefs(seeOregonAdministrativeRule,OAR§581-022-0612):
ExceptionofStudentswithDisabilitiesfromStateAssessmentTesting(1)Forthepurposesofthisrulea"studentwithadisability"isastudentidentifiedundertheIndividualswithDisabilitiesEducationAct,consistentwithOARchapter581,division015,orastudentwithadisabilityunderSection504oftheRehabilitationActof1973.(2)ApublicagencyshallnotexemptastudentwithadisabilityfromparticipationintheOregonStateAssessmentSystemoranydistrictwideassessmentstoaccommodatethestudent'sdisabilityunlesstheparenthasrequestedsuchanexemption.
However,HouseBill2655establishedaStudentBillofRightsonJanuary1,2016,whichpermittedparentsoradultstudentstoannuallyopt-outofOregon'sstatewidesummativeassessments,pursuanttoOAR§581-022-1910.TheGovernorpublishedamemorandumforSuperintendents,Principals,andDistrictTestCoordinatorsrelatedtothechange(seeAppendix1.4.1).Theexpectationthatallstudentsintheassessedgradesparticipate,includingstudentswithdisabilities,iselaboratedclearlyandpervasivelyacrossallguidancedocuments.ForexampleintheOregonTestAdministrationManual(TAM),whereitstatesthat,"Allstudentsenrolledingrades3–8andinhighschoolmusttaketherequiredOregonStatewideAssessmentsofferedattheirenrolledgrade,includingstudentsre-enrolledinthesamegradeasintheprioryear,unlessthestudentreceivesaparent-requestedexemption..."(seeAppendix1.4.2,p.96).
2016-2017ORExtendedAssessment Overview&CriticalElement1–Page24
1.4AEnglishLearnersEnglishlearnersareincludedasappropriateinOregon'sstatewideassessmentsystem.(seeAppendix1.4A.1,pp.31-33).TheSmarterBalancedassessmentdirectionsaretranslatedintomultiplelanguagesandavailableviatheOaksportal.OAR581-022-0620(2)requiresODEtoprovidetranslatedOAKSassessmentsforpopulationsatorabove9%ingradesK-12withinthreeyearsaftertheschoolyearinwhichthelanguageexceedsthethreshold(seeAppendix1.4A.2).Inaddition,theaccommodationsavailabletostudentswhoparticipateintheORExtincludetranslationintothenativelanguage,whereappropriate(seeAppendix2.3A1,pp.36-43).
1.4BNativeLanguageAssessmentsTheORExtisnotadministeredinanativelanguageformat,thoughitcanbetranslatedintoastudent'shomelanguage.1.5ParticipationDataOregon'sparticipationdataindicatethatmoststudentsinthetestedgradelevelsareincludedinourassessmentsystem.Thestudentswithdisabilitiessubgroupdidnotmeetminimumparticipationrequirementsin2015-16,themostcurrentdataavailableatthetimeofthisreport,inEnglishlanguageartsormathematics,withratesat92.3%and91.5%,respectively.Seethetablebelowforasummaryofparticipation.Forcompletereports,seehttp://www.oregon.gov/ode/educator-resources/assessment/Pages/Assessment-Group-Reports-for-2014-2015-and-2015-2016.aspx.DocumentationofthisrequirementisprovidedwithintheAnnualPerformanceReport,IndicatorB3,whichissubmittedtotheUnitedStatesDepartmentofEducation's(USED's)OfficeofSpecialEducationPrograms(OSEP).Participationandperformancesummariesareprovidedbelow.Additionalinformationregardingstateperformanceispublishedinthe2015-16StateReportCard(seeAppendix1.5,pages1-11forstudentandteacherdemographicsandpages21-48forassessmentinformation).
2016-2017ORExtendedAssessment Overview&CriticalElement1–Page25
2016-2017ORExtendedAssessment CriticalElement2–Page26
CriticalElement2–AssessmentSystemOperations2.1TestDesignandDevelopmentThetestspecificationsdocumentthatdescribesourapproachtoassessmentandtestdesignfortheORExtispublishedinAppendix2.1.Thedocumentincludesourapproachtoreducingthedepth,breadth,andcomplexity(RDBC)ofgradelevelcontentstandards,anoverviewoftheessentializationprocessandEAFdocuments,theplannedtestdesignfortheORExt,testdevelopmentconsiderations,sampletestitems,itemspecifications,anduniversaltools/designatedsupports/accommodations.Nonewitemsweredevelopedin2016-17,sothe2014-15testspecificationsarethemostcurrentavailable.
2.1AORExtPurposeThestatedpurposeoftheORExtistoprovidethestatetechnicallyadequatestudentperformancedatatoascertainproficiencyongradelevelstatecontentstandardsforstudentswithsignificantcognitivedisabilities.Along-termgoaloftheprogramistoalsoprovideinformationregardingannualstudentgrowthrelatedtothesecontentstandardsoverGrades3-8,asmeasuredbyverticallyscaledassessmentsinELAandmathematics.Theresultsoftheassessmentarecurrentlyreportedincomparisontofourperformancelevels:Level1,Level2,Level3,andLevel4.Levels3and4denoteaproficientlevelofperformance,whileLevels1and2denoteperformancethatisnotproficient.BRTandODEdevelopedascaledscoreinterpretationguidetoassiststakeholdersininterpretingthemeaningofthescaledscoresgeneratedbytheORExt,supportedbythestate'sachievementleveldescriptors.ThisguidanceispublishedinAppendix2.1A.
2.1BORExtTestBlueprint
Appendix2.1BincludestheentiretestblueprintfortheORExt,asconveyedbythebalanceofrepresentationacrosscontentareasanddomains.Field-testingisconductedeachyearinordertosupportthecontinuousimprovementoftestfunctioning.However,itemsareselectedtomaintainthisbalanceofrepresentation.Oregonteachersvalidatedthecontentoftheassessment,agreeingwiththestandardsthatwereandwerenotselectedtodeveloptheEssentializedStandardstowhichtheORExttestitemsarealigned.
2.1CTestDevelopmentProcessesThetestdevelopmentprocessimplementedfortheORExtisconveyedinAppendix2.1C,includingstandardselectionandvalidation,itemdevelopment,itemreview,reviewofallOregonteacherfeedbackandupdatingofitems,andscalinganditemselection.TheappendixarticulatestheprocessusedtogeneratethematerialswithcommaseparatedvaluefilesusedtocreateitemtemplatesthatfedintoAdobeInDesign©throughadatamerge.FinaltestpackagesarereviewedforaccuracyandcontentandthendisseminatedviasecurefiletransfertoOregonQualifiedAssessors.
2.1DComputer-AdaptiveConsiderationsTheORExtisnotacomputer-adaptiveinstrument,sotheseconcernsdonotapply.
2016-2017ORExtendedAssessment CriticalElement2–Page27
2.2ItemDevelopmentItemwriterswererecruitedbyODEstaffonMay20,2014usinganexistingQualifiedAssessor/QualifiedTrainerlistservprogram,usingthefollowingtext:ProjectDescription:BehavioralResearchandTeachingattheUniversityofOregonisrecruitingOregonteacherstoparticipateinitemdevelopmentforanewalternateassessment,tobefieldtestedinthespringof2015.Selectedteacherswillbeaskedtodevelop360itemsinEnglishLanguageArts,Mathematics,orScienceoverthecourseofthesummer,frommid-JunethroughendofAugust2014.TheProjectDirectorwillworkwithleaditemdeveloperstoprovidetraining,ongoingreviewandfeedback,andqualityassurance.Allparticipantswillbeexpectedtoprovidedocumentationoftheirqualificationsandsigntestsecurityagreements.Inaddition,allitemdeveloperswillbeexpectedtoparticipateinahalf-dayitemdevelopmenttrainingbaseduponthefollowingschedule:
• ELA-Tuesday,June17,2014from8AMto12PM• Math–Wednesday,June18,2014from8AMto12PM• Science–Thursday,June19,2014from8AMto12PM
MinimumQualifications:AlllicensedOregonpublicschoolteacherswithatleastthreeyearsofteachinginalifeskills/severeneedsprogram(SPED)orageneraleducationclassroom(GEN-ED),respectively,areencouragedtoapply.Preferencewillbegivenforitemwritingexperience,additionalyearsofteachingexperience,andhighereducationdegreestatus.Compensation:Teacherswhoparticipateinthisprocesswillbecompensatedatarateof$20/hrviaprofessionalservicecontracts.Itisanticipatedthatteacherswillproduce4ELAitems/hr,6Scienceitems/hr,and8Mathitems/hr.Assuch,themaximumcontractamountforELAwillbe$1,800,forScience$1,440,andforMath$900.Itemdevelopmentwillfocusprimarilyonwritingthestemand3options,withnoneedtoproducegraphics(ratheruselabelsforaBRTgraphicdesignertoproduce).Needs ContentArea GradeLevel(s)2teachers(2SPED) ELA Elementary(G3-5)2teachers(1GEN-ED;1SPED)
ELA Middle(G6-8)
1teachers(1GEN-ED) ELA High(G11)TotalNumberNeededELA 5
1teachers(1SPED) Math Elementary(G3-5)1teachers(1SPED) Math Middle(G6-8)2teachers(1SPED;1GEN-ED)
Math High(G11)
TotalNumberNeededMath 4
2016-2017ORExtendedAssessment CriticalElement2–Page28
3teachers(2SPED;1GEN-ED)
Science G5,8,&11
TotalNumberNeededScience 3
TotalOregonTeacherItemWritersNeeded 12Contact:Ifyoumeettheabovequalificationsandareinterestedinapplyingtoassistonthisproject,pleasecontactDanFarleyatdfarley@uoregon.eduorat541-346-3133.ThedeadlinetoapplyisJune13,2014.Thankyou.Becausethetimelinerequiredworkoverthesummer,Oregonteacherrecruitmentwaschallenging.BRTresearchersthusperformedanadditionalon-campusrecruitmentwithintheCollegeofEducationusingthesameinformation.Thefinalpoolofitemwritersincluded18itemwriters:sevenOregonteachers(allwithMAdegrees),fivePhDcandidateswithintheCOE,andsixBRTresearchers(fourPhDcandidates,onePhD,andonewithanMA).Itemwritersaveraged11.5yearsofteachingexperience.TheteachersrecruitedallhadpriorexperiencedevelopingitemsfortheORExt,asdidalloftheBRTresearchers.ThefivePhDcandidateswithintheCOEhadnoprioritemdevelopmentexperience.AllitemdevelopmentwasreviewedbyBRTresearchersandtheProjectManager.TheitemdevelopmentprocessfollowediselaboratedinAppendix2.2.1,whichisthePowerPointusedintrainingallOregonitemwriters.Theitemdevelopmentprocesswasstructuredwiththefollowingsteps.Itemwriterswerefirstorientedtothestudentpopulation,asthepoolofitemwritersincludedbothcontentandspecialeducationexperts.TheEssentializationProcessusedtoRDBCgradelevelstandardswasthenmodeledsowriterswouldunderstandhowtheitemalignmenttargets,theEssentializedStandards,weregenerated.Lecture,guidedpractice,andindependentpracticeactivitiesandfollow-updiscussionensuredcomprehensionoftheprocess.BRTstaffdevelopedexemplaritemsforeveryEssentializedStandard,varyingthecomplexityfromLow(L)toMedium(M)toHigh(H)levelsofcomplexitytoconveythedifferentperformanceexpectationsateachlevel.Thebalancedverticalscalingdesignprovidedanoverallform-to-formandgrade-to-gradelevelframeworkforthetestformationprocessonceitemsweredeveloped(seeAppendix2.2.2).SampleitemsareprovidedinAppendix2.2.3forstakeholderreference,demonstratingtheformatandstyleoftypicalitemsontheORExt.2.3TestAdministrationTheORExtassessmentsareadministeredaccordingtotheadministration,scoring,analysis,andreportingcriteriaestablishedintheORExtGeneralAdministrationManual(seeAppendix2.3).ImportantupdatestothetestingprocessaredistributedviatheAssessmentandAccountabilityUpdateslistserve,aswell(seehttp://www.oregon.gov/ode/educator-resources/assessment/Pages/Assessment-and-Accountability-Update.aspx).ODEusesthissystemtocommunicateinformationthatisrelevantforthestatewideassessmentsystem,includingtheORExt.AnnouncementsaresenttothelistservbyemailandarealsopostedtotheODEwebsite.ThestandardizationoftestadministrationissupportedbyacomprehensivetrainingprocessdescribedbelowinSection2.3B.
2016-2017ORExtendedAssessment CriticalElement2–Page29
2.3AAdministrationandAccommodationsThestatehasensuredthatappropriateuniversaltools,designatedsupports,andaccommodationsareavailabletostudentswithdisabilitiesandstudentscoveredbySection504byprovidingguidanceandtechnicalsupportonaccommodations(seeAppendices2.3A.1and2.3A.2).Guidelinesregardinguseoftheaccommodationsforinstructionalpurposesareincludedinthedocument,asallstudentsareexpectedtoreceivetestaccommodationsthatareconsistentwithinstructionalaccommodations.AccommodationsarebuiltintotheflexibilityprovidedbytheORExttestthoughtheyhavenotyetbeenresearchedfortheORExt.However,annualtrainingandproficiencytestingeffortsrelatedtobecomingaqualifiedassessorand/orqualifiedtrainerfortheORExtsupportstandardizeduseofavailableaccommodationsthatarenotalreadypartofthetestdesign.Basedonannualanalyses,resultsdemonstratethatstudentperformancevariesaccordingtotheirabilitiesandnotconstruct-irrelevantfactors,suchassex,race,orethnicity(SeeSection4.2).ThestatehasensuredthatappropriateaccommodationsareavailabletostudentswithlimitedEnglishproficiencybyprovidingguidanceandtechnicalsupportonaccommodations(seeAppendix2.3A.1).Communicationsystemsforthisstudentpopulationarelimited;exposuretomultiplelanguagescanmakeastudent'scommunicationsystemmorecomplex.TheORExtusesuniversaldesignprinciplesandsimplifiedlanguageapproachesinordertoincreaselanguageaccesstotestcontentforallstudents.Inaddition,directionsandpromptsmaybetranslated/interpretedforstudentsintheirnativelanguage.Ananalysisofaccommodatedversusnon-accommodatedadministrationsisneededinordertodemonstratethattheprovisionoflanguageaccommodationsisnotprovidinganyadvantagetostudentswithlimitedEnglishproficiency,noranydisadvantagetootherparticipants.Accommodationsinformationwascollectedthisyearasanoptionfordataentry.Enteringaccommodationsinformationwillberequirednextyear.AnalysesoftheimpactofaccommodationprovisionontheORExtshouldthusbefeasibleafterthespring2018administration.TheOregonExtendedassessmentscanbeadministeredusingbothLargePrintandBraille(contractedandnon-contracted)versions,aswell.OregonhasensuredthattheOregonExtendedassessmentsprovideanappropriatevarietyofaccommodationsforstudentswithdisabilities.Thestatehasprovidedguidanceonaccommodationsinpresentation,response,setting,andtimingintheAccommodationsManual2013-14:HowtoSelect,Administer,andEvaluateAccommodationsforOregon'sStatewideAssessments(seeAppendix2.3A.2).TheOregonExtendedassessmentsarealsodesignedaccordingtouniversaldesignprinciplesandutilizeasimplifiedlanguageapproach(seeAppendix2.3A.3).Inthe2013-2014schoolyear,thestatedevelopedatrainingandproficiencyprogramforsignlanguageinterpretationofitsassessmentsandhasupdatedthesiteannuallysincethattime.Thetrainingprocess(http://lms.brtprojects.org)includedvideosofinterpretersadministeringitemstostudents,materialsthatsupportappropriateadministration(i.e.,
2016-2017ORExtendedAssessment CriticalElement2–Page30
transcriptsandPowerPointslidesthatsupplementthevideoadministrationsandthecurrentODEaccommodationsmanual),andproficiencytestingtosupportstandardizedinterpretationforOregon'sassessments,includingtheORExt.A10-itemproficiencytestwasadministered,withan80%requiredforpassing(8/10itemscorrect).In2016-17,thesitewasusedtotrain60participants.Allparticipantspassedtheassessmentonthefirstattempt.Theoverallaveragescoreontheproficiencytestwas97.6%.TheORExtassessmentsprovideanappropriatevarietyoflinguisticaccommodationsforstudentswithlimitedEnglishproficiency.Theyalsouseasimplifiedlanguageapproachintestdevelopmentinordertoreducelanguageloadofallitemssystematically(seeAppendix2.3A3).Anygivenstudent'scommunicationsystemmayincludehomesigns,schoolsigns,Englishwords,andSpanishwords,forexample.Withtheexceptionofitemsthatrequireindependentreading,theORExtassessmentcanbetranslatedorinterpretedbyaQualifiedAssessor(QA)workingwithaninterpreterinthestudent'snativelanguage,includingAmericanSignLanguage.QAsareallowedtotranslate/interpretthetestdirections.QAscanadapttheassessmenttomeettheneedsofthestudent,whilestillmaintainingstandardizationduetosystematicpromptsandwell-definedanswers.
2.3BComprehensiveTrainingSystemComprehensiveinformationforongoingtrainingforallqualifiedassessors(QAs)andQualifiedTrainers(QTs)isprovidedinAppendices2.3B.1-2.3B.8.TrainingandQA/QTproficiencyisdeterminedannuallyviaanonlinedistributionandassessmentsystemlocatedathttps://or.k12test.com.Thiswebsitehostsallresourcesandinformationneededtoadminister,score,report,andinterprettheresultsfromtheORExt.ThewebsitealsoincludesproficiencyassessmentsthatarerequiredforallQAsandQTswhomayadministertheORExt.QTsaredirectlytrainedbyODEandBRTstaffaspartofatrainthetrainersmodel.QTsthenprovidedirecttrainingsfornewQAsintheirrespectiveregions.TheOregonDepartmentofEducation(ODE)providedfourdirectstatewidetrainingsfornewQualifiedTrainers(QTs)andQualifiedAssessors(QAs)inface-to-faceregionaltrainings.Theschedulefortheregionaltrainings,aswellasrelevanttraininginformation,isprovidedbelow:
2016-2017ORExtendedAssessment CriticalElement2–Page31
Date Who/Team Location
11-1-2016
Team:BradLenhardt,GeraldTindal,&SevrinaTindalContact:MaryApplemary.apple@imesd.k12.or.us
IMESDPendleton,OR
11-3-2016
Team:BradLenhardt,GeraldTindal,&SevrinaTindalContact:CatherineHalliwell-TemplinCatherine.halliwell-templin@hdesd.org
HDESD-Redmond,OR
11-10-2016Team:BradLenhardt&DanFarleyContact:PamWurzellpam_wurzell@soesd.k12.or.us
SOESD-Medford,OR
11-15-2016Team:BradLenhardt&DanFarleyContact:SharonMeeuwsensharon_m@nwresd.k12.or.us
NWESDHillsboro,OR
11-17-2016
Team;BradLenhardt&DanFarleyContact:EleniBostoneleni.boston@wesd.org WillametteESD
Salem,OR
OnlytrainedQualifiedAssessors(QAs)canadministertheOregonExtendedassessment.QualifiedAssessorswhoalsoreceivedirectinstructionfromODEandBRTmaybecomeQualifiedTrainers(QTs)whoarecertifiedtotrainlocalstaffusingthetrain-the-trainersmodel.Trainingfornewassessorsmustbecompletedonanannualbasis.Assessorswhodonotmaintaintheirrespectivecertificationsforanygivenyearmustre-trainiftheychoosetoenterthesystemagain.ThetablesbelowcontaindatafromtheOregonExtendedAssessmentTrainingandProficiencyWebsite(http://or.k12test.com/).AllassessorsneedtocompletesomeformoftrainingeachyeartoretaintheirstatusforadministeringtheExtendedAssessments.Newassessorsandreturningassessorswhoneededfurthertrainingin2016-17wererequiredtopassfourproficiencieswithascoreof80%orhigher.ThesefourproficiencieswereinAdministration,EnglishLanguageArts(ELA),Mathematics,andScience.ReturningQAsorQTsforthe2016-17schoolyearonlyneededtopassaRefresherProficiency,againwithascoreof80%orhigher.Thetablesbelowcontaindataonthenumberofassessors(participants)ineachofthefourproficiencies,aswellastheRefresherProficiency.Includedinthedataisthenumberofattemptsneededtoattainapassingscoreaswellastheaveragepassingscoreoftheparticipants.
2016-2017ORExtendedAssessment CriticalElement2–Page32
AnanalysisoftheOregonExtendedAssessmentTrainingandProficiencyWebsiteshowed353Assessorsin-Training,1,030QualifiedAssessors,and137QualifiedTrainers.322TestParticipants–AdministrationProficiency
NumberofParticipants
PercentageofParticipants AttemptstoPass AveragePassing
Score270 83.9% 1 91.5%44 13.7% 2 90%6 1.9% 3 84%1 <1% 4 0%1 <1% 5 85%
268TestParticipants–EnglishLanguageArtsProficiency
NumberofParticipants
PercentageofParticipants AttemptstoPass AveragePassing
Score259 96.6% 1 95.7%9 3.4% 2 87.8%
264TestParticipants–MathematicsProficiency
NumberofParticipants
PercentageofParticipants AttemptstoPass AveragePassing
Score259 98.1% 1 96.6%4 1.5% 2 90%1 <1% 3 100%
259TestParticipants–ScienceProficiency
NumberofParticipants
PercentageofParticipants AttemptstoPass AveragePassing
Score257 99.2% 1 96.9%1 <1% 2 0%1 <1% 3 100%
936TestParticipants–RefresherProficiency
NumberofParticipants
PercentageofParticipants AttemptstoPass AveragePassing
Score914 97.6% 1 93.7%22 2.4% 2 91.6%2 <1% 3 96%
AhighernumberofassessorscompletedtheRefresherProficiencytestthanthesubjectareaproficiencytestsreflectingagreaternumberofreturnassessorscomparedtonewassessors.AdministrationProficiencycontinuedtobethemostchallengingtonewassessors,butmostwereabletopassonthefirstorsecondattemptwithlessthan2%ofassessorsrequiringmorethantwoattempts.ThemajorityofassessorspassedtheELA,
2016-2017ORExtendedAssessment CriticalElement2–Page33
Math,Science,andtheRefresherproficiencytestsonthefirstattemptwithlessthan4%requiringasecondorthirdattempt.Therewere73fewerQualifiedAssessorsbut8moreQualifiedTrainerscomparedtolastyear.EvaluationsarecollectedateachQTtraininginNovember.Theresultsreflectgeneralapproval,butalsosuggestareasofimprovementthatODEandBRTworkonforsubsequenttrainings/subsequentyears,asappropriate.QTevaluationsthisyearincludedpositivelywordedstatementsregardingthequalityoftrainingratedonascalewhere1=StronglyDisagree,2=Disagree,3=Agree,and4=StronglyAgree.Thefirstsectionevaluatedthestate-levelinformationandtheknowledgeoftheODEpresenters,theparticipants’levelofcomfortwiththetrainingprovided,theparticipants’abilitytocarrythistrainingandmaterialsbacktotraindistrictstaff,andtheoverallutilityofthetraining.Seventy-eightpercentofparticipantsstronglyagreedwiththesestatements,19%agreed,andlessthan3%disagreedandstronglydisagreed,collectively.Inthesecondsection,participantswereaskedtoevaluatetheBRTtrainersandtheirguidelinesregardinghowtousethetrainingandproficiencywebsiteandrelatedresources.Seventy-ninepercentofparticipantsstronglyagreedwiththesestatements,19%agreed,andlessthan2%disagreedandstronglydisagreed,collectively.Overall,theseresultsdemonstratethatparticipantsfeltthatthetrainingwashighqualityandtheyfeltconfidentthattheycouldtraintheirstaffuponreturntotheirrespectivedistrictswiththeknowledgeandresourcesgained.Thisyear’sQTtrainingcycleincludedanoptionalafternoonsessionforanyinterestededucatorsonhowtoessentializegradelevelcontentstandardsandhowtodevelopcurriculumandprovideinstructionthatisalignedtothosestandardsforstudentswhoarefunctioningoffgradelevel,withafocusonstudentswithsignificantcognitivedisabilities(SWSCD).Weaskedparticipantstoratetheirconfidenceinusingtheknowledgeacquiredduringthesessionaswellastoevaluatethequalityofthepresentationandmaterials.Afour-pointscalewasemployed(StronglyDisagree,Disagree,Agree,StronglyAgree).ThesurveyfortheafternoonsessionwasconductedonlinewithQualtricssoftware.Percentagesofresponsesforeachstatementusedinthesurveyareprovidedbelow.Thefirsttableprovidesasummaryofthedatarelatedtoparticipantconfidence,whilethesecondprovidestheirevaluationofthequalityofthepresentation.Therespondentn-sizesrangedfrom26-30,dependinguponthequestion.Abargraphofstudyresultsisprovidedbelow,followedbytablesofdescriptivestatistics.Thedatavisualizationbelowwasconductedwithggplotinthetidyversepackage(Wickham,H.,2017).
2016-2017ORExtendedAssessment CriticalElement2–Page34
Note:Thefirsttwographspresentparticipants’confidenceintheirabilitytousetheinformationpresentedintheareasofessentializationandcurriculumandassessment,respectively.Thenextfourgraphsconveyevaluationsofthepresentersforthecurricularandinstructionandessentializationtrainings.Resultsareverypositive,withsomereviewersfeelinglessconfidentabouttheirabilitiestotrainothersabouttheessentializationprocess.Thisoutcomewasexpected.Theprocessiscomplex,particularlygiventheunderstandingthatthiswasthefirsttimetheyhadreceivedsuchtraining.
Confidence_CI Confidence_Ess Eval_CI_SWSCD Eval_Ess Eval_GP_CI_SWSCD Eval_GP_Ess
0
20
40
60
Ratings by Group
Ave
rage
Gro
up R
atin
gs
ratingStrongly Agree
Agree
Disagree
Strongly Disagree
2016-2017ORExtendedAssessment CriticalElement2–Page35
ConfidenceScalePercentagesStatementFollowingthecurriculumandinstructiontraining,Iamconfident:
StronglyDisagree Disagree Agree Strongly
Agree
1. Inmyunderstandingoftheessentializationofgradelevelacademiccontentstandards.
0 0 60 40
2. InmyabilitytoessentializetargetedELA,Math,orSciencecontentstandards.
0 10 56 33
3. Inmyabilitytoessentializecontentstandardssuchthattheyarealignedtoastudent’scurrentlevel(s)ofperformance.
0 6 63 30
4. InmyabilitytoaccessandusetheCurricularandInstructionalMaterialsforStudentswithSignificantCognitiveDisabilitiesonlinetrainingcourseandresources.
0 6 73 20
5. Inmyabilitytotrainothersintheessentializationofgradelevelcontentstandardsprocess.
0 31 51 17
6. InmyunderstandingoftheavailableC&Iresourcesandhowtoaccessthem.
0 3 65 30
7. InmyabilitytodevelopC&IlessonsinELAand/orMaththatarealignedtoEsSt.
0 3 76 19
2016-2017ORExtendedAssessment CriticalElement2–Page36
StatementFollowingthecurriculumandinstructiontraining,Iamconfident:
StronglyDisagree Disagree Agree Strongly
Agree
8. InmyabilitytodevelopC&Ilessonsatmultiplelevelsofcomplexity.
0 3 76 19
9. ThatIcouldexplaintopeersandcolleagueshowtoaccessandusetheavailableC&IresourcesforSWSCD.
0 3 76 19
10. InmyabilitytoessentializestandardsanddevelopPLAAFP,IEPGoalsandObjectives,andC&IlessonsthatarealignedtothetargetedEsSt.
0 7 76 15
Note.ELA=Englishlanguagearts;C&I=CurriculumandInstruction;SWSCD=StudentswithSignificantCognitiveDisabilities;PLAAFP=PresentLevelsofAcademicAchievementandFunctionalPerformance;IEP=IndividualizedEducationProgram.Evaluation–EssentializationProcessandIntendedUsesPercentagesStatement
StronglyDisagree Disagree Agree Strongly
Agree1. Theinformationforthissectionofthetrainingwaswellorganizedandeasytofollow.
0 3 57 38
2. Thepresenter(s)wereknowledgeable,organized,andclear.
0 0 50 50
3. Thepresenterengagedparticipantstodiscussandapplywhatwewerelearning.
0 0 57 42
4. ThetrainingprovidedandreferencedresourcesthatIcanuseafterthesession.
0 0 42 57
2016-2017ORExtendedAssessment CriticalElement2–Page37
Statement
StronglyDisagree Disagree Agree Strongly
Agree5. Ifeltcomfortablewiththeinformationandresourcespresented.
0 3 50 46
GuidedPracticeEssentializingStandards6. Theinformationforthissectionofthetrainingwaswellorganizedandeasytofollow.
0 3 53 42
7. Thepresenter(s)wereknowledgeable,organized,andclear.
0 3 53 42
8. Thepresenterengagedparticipantstodiscussandapplywhatwewerelearning.
0 0 61 38
9. ThetrainingprovidedandreferencedresourcesthatIcanuseafterthesession.
0 0 50 50
10. Ifeltcomfortablewiththeinformationandresourcespresented.
0 3 57 38
Note.ELA=Englishlanguagearts;C&I=CurriculumandInstruction;SWSCD=StudentswithSignificantCognitiveDisabilities;PLAAFP=PresentLevelsofAcademicAchievementandFunctionalPerformance;IEP=IndividualizedEducationProgramEvaluation–CurricularandInstructionalResourcesforSWSCDPercentagesStatement
StronglyDisagree Disagree Agree Strongly
Agree1. Theinformationforthissectionofthetrainingwaswellorganizedandeasytofollow.
0 0 53 46
2. Thepresenter(s)wereknowledgeable,organized,andclear.
0 0 50 50
3. Thepresenterengagedparticipantstodiscussandapplywhatwewerelearning.
0 0 53 46
2016-2017ORExtendedAssessment CriticalElement2–Page38
Statement
StronglyDisagree Disagree Agree Strongly
Agree4. ThetrainingprovidedandreferencedresourcesthatIcanuseafterthesession.
0 0 53 46
5. Ifeltcomfortablewiththeinformationandresourcespresented.
0 0 53 46
GuidedPracticeEssentializingStandards6. Theinformationforthissectionofthetrainingwaswellorganizedandeasytofollow.
0 0 50 50
7. Thepresenter(s)wereknowledgeable,organized,andclear.
0 0 50 50
8. Thepresenterengagedparticipantstodiscussandapplywhatwewerelearning.
0 0 57 42
9. ThetrainingprovidedandreferencedresourcesthatIcanuseafterthesession.
0 0 50 50
10. Ifeltcomfortablewiththeinformationandresourcespresented.
0 0 57 42
Note.ELA=Englishlanguagearts;C&I=CurriculumandInstruction;SWSCD=StudentswithSignificantCognitiveDisabilities;PLAAFP=PresentLevelsofAcademicAchievementandFunctionalPerformance;IEP=IndividualizedEducationProgramInaddition,alltechnicalassistancequestionsthatwereceivefromthefieldaspartofourHelpDeskaredocumented.Thelogofthetechnicalassistanceprovisionisreviewedeachmonth,aswellasannually,inordertodeterminewhataspectsofourassessmentsystemneedfurtherclarificationorimprovement.TheHelpDesklogispublishedinAppendix2.3B.9.Oregonmonitorsthequalityofitssysteminseveralwaysinordertosupportcontinuousimprovement.Intermsoftheassessmentquality,itemstatisticsarereviewedeachyearanditemsthatarenotfunctioningasintendedareremovedandreplacedbybetterfunctioningfield-testitems.In2014-15,itemswerereviewedintwophases,firstusingclassicaltesttheory(CTT)and
2016-2017ORExtendedAssessment CriticalElement2–Page39
secondusingRaschanalyses.Allitemsflaggedasaresultofthestatisticalreviewswereanalyzed,item-by-item,byateamofmeasurementandcontentexpertsatBRT.Notallflaggeditemswereremoved,asseveraldidnothaveapparentdesignflaws.Considerationsregardingdomainrepresentationaswellasitemdifficultyrangealsowereconsideredduringthereviewprocess.Wealsoemployeddifferentdecisionrulesforuniqueitemsversushorizontally-orvertically-scaledanchoritems.Itwasimportantinmanycasestomaintainanchoritems.Itemswithcleardesignflawswereremovedfromsubsequentanalysesandreporting.Thefollowingflaggingcriteriawereemployed:
CTT:Auniqueitemwasflaggedifithadap-valueof.10orlower,.90orhigher,orapointbiserial<.15.Anchoritemswereflaggediftheyhadap-valueof.10orloweror.95andhigheronallformsorapointbiserial<.45onanyform.Rasch:Uniqueitemswereflaggediftheiroutfitmeansquarevalueswerebetween0and.25or>1.5.Anchoritemswereflaggediftheiroutfitmeansquarevalueswere<.5,>1.8forhorizontalitems,or>2.0forverticalanchoritems.
Outofatotalof5,929itemsdevelopedin2014-15,166wereremoved(2.8%).WealsoimplementaconsequentialvaliditystudyeachyearthatsurveysQAsandQTsregardingtheacademicandsocialconsequencesoftheORExt,bothintendedandunintended.TheConsequentialValidityreportispublishedinAppendix2.3B.10.ODEandBRTstaffreviewtheresultsofthesurveyannuallytodeterminewhatprogramimprovementsareneeded.Asummaryoftheresultsisprovidedbelow.ODEimplementedaresearchsurveyprogramtoaddresstheneedtodocumenttheconsequences,bothintendedandunintended,oftheORExtAssessments.Theresearchquestionshavebeenframedbaseduponcurrentconsequentialvalidityapproachesforalternateassessmentsintheliterature,aswellasissuesthatareofspecificvalueinOregon.Thesurveyincluded344respondents.Thiswas25%ofthesolicitedrespondents,whowereallQualifiedAssessors(QAs)andQualifiedTrainers(QTs)intheor.k12test.comdatabase.Thesamplewas84%femaleandrepresentedallregionsofthestate,aswellasageranges.Thesurveyincludedarangeofquantitativeandqualitativecomponents.ThequantitativeresultsdemonstratethatQAsandQTscontinuetofeelthattheORExttestitemswereeasytoadministerandscore(58%StronglyAgree)andfeltconfidentintheirabilitytointerpretscaledscoresandAchievementLevelDescriptorsfortheORExt(54%Agree).Theyalsofeltthattheitemswereaccessibleforstudentswhoparticipated(53%Agree)andthattheORExtreflectedtheacademiccontentthatSWSCDshouldbelearning(57%Agree).QAsandQTsfeltmarginallypositiveabouttheeducationalimpactsoftheORExtandmarginallynegativeaboutitssocialimpacts.TheresultsagaindemonstratethattheORExtcontentareaassessmentsgenerallyrequireuptoonehourtoadminister.ThequalitativeresultsrevealedtwoareasinwhicheducatorsappreciatedtheORExtandfourareasofneededimprovement.QAsandQTssaidthattheyappreciated:1)theassessment'sefficiency(i.e.,morestreamlinedadministration,easeofadministration,easiertogiveandscoreonline,onlinematerialsdistribution);and,2)overallitemandtestdesign(i.e.,oneitemperpage,visualsupports,scoringprotocolandstudentmaterials
2016-2017ORExtendedAssessment CriticalElement2–Page40
design,accessibilityoftestquestions).Teachersrecommendedthefollowingareasofimprovement,notallofwhichareactionable:1)Optiontoadministertheassessmentelectronically,2)Afunctionalskillsassessment,3)Newitemsforverylowfunctioningstudentsshouldbedeveloped,and4)Amathassessmentcomposedofmorepractical/lifeskillsproblemsinvolvingtimeandmoney.Completeresults,includinganticipatedresponses,fromthesurveycanbefoundinAppendix2.3B.10.
2.3CTechnology-basedAssessmentsTheORExtwasimplementedonasmallscaleusingatechnology-basedplatformaspartofPhase2oftheORExtPilotTabletAdministrationstudyconductedinthespring2017.Thecurrentplanistomakethetablet-basedadministrationoftheORExtavailablestatewidenextyear,2017-18.Acompletereportoftheresultsofthestudy,includingtheresearchplanandthelessonslearned,isprovidedinAppendix2.3C.Inshort,thisyear’stabletadministrationstudydemonstratedthatQAssupportatabletadministrationoftheORExtatthestatewidelevel.Thestudyresultsalsodemonstratedthatadditionaltrainingmustbeprovidedforthemanualwritingscoringprocess.Inaddition,theadministrationoftheORoraforstudentswhoseORExttestingisdiscontinuedaftertheyhavemettheminimumparticipationrulewillbeincorporatedintothetabletadministrationnextyear.Tosupporttrainingandunderstandingofthesystembybothteachersandstudents,practiceitemsinatabletformatwillbeprovidedforstakeholderstoprepareforfuturetablet-basedadministrations.Thisyear’sstudyaddressedissuesrelatedtodatabasecommunicationsystemstoensuredatasecurityandaccuratedatastorageandaccessatthedistrictlevel.Thepaper/pencilversionwillcontinuetobeavailableforstudentswhocannotaccessatabletadministration.2.4MonitoringTestAdministrationTheODEmaintainsarigoroustrainingsystemtosupportstandardizedtestadministrationfortheORExt,locatedathttps://or.k12test.com(securewebsite,butseescreenshotbelowforanexampleoftrainingcontent).
2016-2017ORExtendedAssessment CriticalElement2–Page41
Theor.k12test.comwebsiteincludesatrainingsectionthataddressesanysystemsupdates,theprocessforbecomingaQualifiedAssessororQualifiedTrainer,studenteligibilityexpectations,studentconfidentialityandtestsecurity,testadministrationandscoringexpectations,examplesofappropriateandinappropriateadministration(video),supportingstudentaccesstoitemswithoutviolatingthetestconstruct,contentareatrainingsthatdemonstratehowtoadministeritemsinELA,Math,andScience(video,withsupportingtestmaterials),andhowtoaccesssecuretestsandcompletedataentry.InformationforQAs,QTs,andparentsregardingtheORExtisalsoprovided,asareallnecessarysupportmaterials.ForQAs,thesematerialsincludepracticeteststoprepareboththemselvesandstudentsfortheannualassessmentandallofthetrainingmaterialsusedonthewebsite.Inadditiontothesematerials,QTshaveaccesstoalltrainingmaterialsnecessarytoprovideannualtrainingtoQAsintheirpurview(seescreenshotbelow):
2016-2017ORExtendedAssessment CriticalElement2–Page42
Inaddition,monitoringandreportingrelatedtotestadministrationissuesfortheORExtisaddressedviageneralODEreportingsystems.InformationregardingthisprocesscanbelocatedinthegeneralassessmentsystemPeerReviewevidencesubmission.2.5TestSecurity
2.5APreventionofAssessmentIrregularitiesTestsecuritypoliciesandconsequencesforviolationareaddressedintheTestAdministrationManualonanannualbasis(seeAppendix1.4.2,p.29-33).Thesepoliciesincludetestmaterialsecurity,propertestpreparationguidelinesandadministrationprocedures,consequencesforconfirmedviolationsoftestsecurity,andannualtrainingrequirementsatthedistrictandschoollevelsforallindividualsinvolvedintestadministration.Consequencesforadult-initiatedtestirregularitiesmaybesevere,includingplacingteachinglicensesinjeopardy(seeAppendix1.4.2,p.31-33).
2016-2017ORExtendedAssessment CriticalElement2–Page43
2.5BDetectionofTestIrregularitiesTheODEutilizesalocalizedmonitoringsystemwhereschooltestcoordinatorsoverseebuilding-leveladministrationbytrained,QualifiedAssessors,andreporttocentralizeddistricttestcoordinators,whoarethenresponsibleforreportinganyconfirmedviolationstoODE.Improprietiesaredefinedasadult-initiatedorstudent-initiatedandinvestigatedaccordingly(seeAppendix1.4.2,p.29-31).
2.5CRemediationFollowingTestSecurityIncidentsODE'salternateassessmentprogrammanagerinvestigatesandremediatessubstantiatedtestsecurityincidentsfortheORExtbyworkingwithdistricttestcoordinators.AdditionalinformationregardingthisprocesscanbelocatedinthegeneralassessmentsystemPeerReviewevidencesubmission.
2.5DInvestigationofTestIrregularitiesSchoolanddistricttestcoordinatorsconductinitialinvestigationsintoallallegedtestirregularities.OncereportedtoODE,allallegedtestirregularitiesareinvestigatedinconsultationwithdistricttestcoordinatorsandthetestvendor,asappropriate(seeAppendix1.4.2,p.31-33).Intheeventthatatestirregularityisdeterminedtobefactual,consequencesaredeterminedbaseduponcontextualissuesthatarebroughttolightduringtheinvestigation.AdditionalinformationregardingthisprocesscanbelocatedinthegeneralassessmentsystemPeerReviewevidencesubmission.2.6SystemsforProtectingDataIntegrityandPrivacy
2.6AIntegrityofTestMaterialsTestmaterialsfortheORExtaremaintainedthroughoutdevelopment,dissemination,andadministrationviamultiplemechanisms.AllitemsunderdevelopmentarestoredinsecurefileserversmanagedbyBehavioralResearch&TeachingattheUniversityofOregon,thetestvendorfortheORExt.Itemreviewsnecessarytoprovidealignment,bias,andsensitivityinformationareconductedonlineusingthesecureDistributedItemReview(DIR)platformhttp://brtitemreview.com(securewebsite,butseeAppendix3.1Bforasystemoverview).Forthe2016-2017schoolyear,allsecuretestdistributionanddataentrywashostedbyODE'ssecurefiletransfersystem,whichisapassword-protectedtestdistributionanddataentrysystemlocatedathttps://district.ode.state.or.us/apps/login/(securewebsite,butseescreenshotbelowforreference).AdataentryguideisprovidedinAppendix2.6.
2016-2017ORExtendedAssessment CriticalElement2–Page44
AdditionalinformationregardingtestsecuritycanbelocatedinthegeneralassessmentsystemPeerReviewevidencesubmission.
2.6BSecureStudent-LevelAssessmentDataStudentleveldataisprotectedbyrelevanttrainingandthroughasecuredatasysteminwhichalldataentryisconductedonlineusingpassword-protected,secureproceduresonthehttps://or.k12test.comorhttps://district.ode.state.or.us/apps/login/websites,asidentifiedabove.Onlytraineduserswithavestededucationalinterestwhohavesignedtestsecurityagreementsareauthorizedtoaccesstoonlinedataentrysystems.SeeAppendix2.6foradditionaldataentryexpectationsfor2016-17.
2016-2017ORExtendedAssessment CriticalElement2–Page45
2.6CProtectingPersonallyIdentifiableInformationAllconfidential,personallyidentifiablestudentinformationisprotectedbypolicyandsupportedbytraining(seeAppendix1.4.2,p.26).Theminimumnumberofstudentsnecessarytoallowreportingofstudentsandstudentsubgroupsvariesbyrating(i.e.,achievement,growth,graduation,andschoolsize)bylevel(i.e.,school/district/state),andbynumberofyearsofassessmentdataavailable.Forexample,toreceiveanachievementrating,schoolsmusthaveatleast40testsforthetwomostrecentschoolyearsinreadingormathematics.Alternatively,smallschoolsreceiveanachievementratingiftheyhaveatleast40testsoverthemostrecentfouryears.Ifaschooldoesnothaveatleast40testsoverafour-yearperiod,theywillnotreceiveanachievementscore(seeAppendix2.6C).Similarrulesareappliedtostudentsubgroups,includingstudentswithdisabilities,Englishlearners,andstudentsfromdiverseracial/ethnicbackgrounds(seeAppendix2.6C,p.7).
2016-2017 OR Extended Assessment Critical Element 3 – Page
46
CriticalElement3–TechnicalQuality:Validity3.1OverallValidity,IncludingValidityBasedonContentAselaboratedbyMessick(1989)1,thevalidityargumentinvolvesaclaimwithevidenceevaluatedtomakeajudgment.Threeessentialcomponentsofassessmentsystemsarenecessary:(a)constructs(whattomeasure),(b)theassessmentinstrumentsandprocesses(approachestomeasurement),and(c)useofthetestresults(forspecificpopulations).Validationisajudgmentcallonthedegreetowhicheachofthesecomponentsisclearlydefinedandadequatelyimplemented.Validityisaunitaryconceptwithmultifacetedprocessesofreasoningaboutadesiredinterpretationoftestscoresandsubsequentusesofthesetestscores.Inthisprocess,wewantanswersfortwoimportantquestions.Regardlessofwhetherthestudentstestedhavedisabilities,thequestionsareidentical:(1)Howvalidisourinterpretationofastudent'stestscore?and(2)Howvalidisittousethesescoresinanaccountabilitysystem?Validityevidencemaybedocumentedatboththeitemandtotaltestlevels.WeusetheStandards2(AERAetal.,2014)indocumentingevidenceoncontentcoverage,responseprocesses,internalstructure,andrelationstoothervariables.Thisdocumentfollowstheessentialdatarequirementsofthefederalgovernmentasneededinthepeerreviewprocess.3ThecriticalelementshighlightedinSection4inthatdocument(withexamplesofacceptableevidence)include(a)academiccontentstandards,(b)academicachievementstandards,(c)astatewideassessmentsystem,(d)reliability,(e)validity,and(f)otherdimensionsoftechnicalquality.Inthistechnicalreport,dataarepresentedtosupporttheclaimthatOregon’sAA-AAASprovidesthestatetechnicallyadequatestudentperformancedatatoascertainproficiencyongradelevelstatecontentstandardsforstudentswithsignificantcognitivedisabilities–whichisitsdefinedpurpose.TheAA-AAASarelinkedtogradelevelacademiccontent,generatereliableoutcomesatthetestlevel,includeallstudents,haveacogentinternalstructure,andfitwithinanetworkofrelationswithinandacrossvariousdimensionsofcontentrelatedtoandrelevantformakingproficiencydecisions.SampleitemsthatconveythedesignandsamplecontentofORExtitemsareprovidedinAppendix2.2.3.Theassessmentsareadministeredandscoredinastandardizedmanner.AssessorswhoadministertheORExtaretrainedtoprovidethenecessarylevelofsupportforappropriatetestadministrationonanitem-by-itembasis.Therearefourlevelsofsupportoutlinedintraining:fullphysicalsupport,partialphysicalsupport,promptedsupport,andnosupport.Itemsweredesignedtodocumentstudents’skillandknowledgeongradelevelacademiccontentstandards,withthelevelofsupportprovideddesignednottointerferewiththe
1Messick,S.(1989).Validity.InR.L.Linn(Ed.),Educationalmeasurement(3rded.,pp.13-103).NewYork:
AmericanCouncilonEducation.2AmericanEducationalResearchAssociation(AERA),AmericanPsychologicalAssociation,&National
CouncilonMeasurementinEducation(2014).Standardsforeducationalandpsychologicaltesting.Washington,DC:AERA.
3U.S.DepartmentofEducation(2015).PeerReviewofStateAssessmentSystems:Non-RegulatoryGuidanceforStatesforMeetingtheRequirementsoftheElementaryandSecondaryEducationActof1965,asAmended.
2016-2017 OR Extended Assessment Critical Element 3 – Page
47
constructbeingmeasured.OnlyonetestadministrationtypeisusedfortheORExt,patternedaftertheformerScaffoldversionoftheassessment.Assessorsadministerthepromptandifthestudentdoesnotrespond,theAssessorreadsadirectivestatementdesignedtofocusthestudent'sattentionuponthetestitemandthenrepeatstheprompt.Ifthestudentstilldoesnotrespond,theAssessorrepeatsthepromptasneededandotherwisescorestheitemasincorrectandmovesontothenextitem.TrainingdocumentationisprovidedinAppendices2.3B.1-2.3B.8.Giventhecontent-relatedevidencethatwepresentrelatedtotestdevelopment,alignment,training,administration,scoring,thereliabilityinformationreflectedbyadequatecoefficientsfortests,and,finally,therelationoftestsacrosssubjectareas(providingcriterion-relatedevidence),weconcludethatthealternateassessmentjudgedagainstalternateachievementstandardsallowsvalidinferencestobemadeonstateaccountabilityproficiencystandards.
3.1AAlignmentBetweenAA-AAASandAcademicContentStandards
OurfoundationofvalidityevidencefromcontentcoveragefortheORExtcomesintheformoftestspecifications(seeAppendix2.1)andtestblueprints(seeAppendix2.1B).Amongotherthings,theStandards(AERAetal.,2014)2suggestspecificationsshould“definethecontentofthetest,theproposedtestlength,theitemformats…”(Standard4.2,p.85).2Allitemsarelinkedtogradelevelstandardsandaprototypewasdevelopedusingprinciplesofuniversaldesign4withtraditional,content-referencedmultiple-choiceitemwritingtechniques5.Themostimportantcomponentintheseinitialstepsaddressedlanguagecomplexityandaccesstostudentsusingbothreceptive,aswellasexpressive,communication.Additionally,bothcontentbreadthanddepthwereaddressed.WedevelopedonetestformfortheORExtthatutilizesascaffoldapproach.Thisapproachallowsforstudentswithverylimitedattentiontoaccesstestcontent,whilethesupportsarenotutilizedforstudentswhodonotneedthissupport.Wedevelopedthetestiterativelybydevelopingitems(seeAppendix2.2.1,whichconveysouritemwritertrainingmaterials),pilotingthem,reviewingthem,andeditingsuccessivedrafts.WeusedacombinationofexistingpanelsofveteranteacherswhohaveworkedwiththeOregonDepartmentofEducation(ODE)invariousadvisingrolesontestingcontentingeneralandspecialeducation,usingthesameprocessesandcriteria,aswellastheintroductionofnewerteacherswhoarequalifiedasweproceedtoremainrelevant.BehavioralResearchandTeaching(BRT)personnelconductedtheinternalreviewsofcontent.Aftertheinternaldevelopmentofprototypeitems,allreviewstheninvolvedOregoncontentandspecialeducationexpertswithsignificanttrainingandK-12classroomexperience.
4Johnstone,C.,Thompson,S.,Bottsford-Miller,N.,&Thurlow,M.(2008).Universaldesignandmultimethodapproachestoitemreview.EducationalMeasurement,27(1),26-36.5Halydyna,T.,&Rodriquez,M.C.(2013).Developingandvalidatingtestitems.NewYork,NY:Routledge.
2016-2017 OR Extended Assessment Critical Element 3 – Page
48
TheORExtincorporatescontinuousimprovementintoitstestdesignviafield-testinginallcontentareasonanannualbasis,withanaverageof25%newitems.Theseitemsarecomparedtooperationalitemsbasedonitemfunctioningandtestdesignfactors,generatingdatausedtoreplaceitemsonanannualbasis,incorporatingthenewitemsthatfillaneededgapwithregardtocategoricalconcurrence,orprovideforawiderrangeoffunctioningwithregardtocomplexitylevels:low–medium–high,comparabletoWebb’sDOK(seeSection3.1A).BRTemployedamulti-stagedevelopmentprocessin2014-15toensurethattestitemswerelinkedtorelevantcontentstandards,wereaccessibleforstudentswithsignificantcognitivedisabilities,andthatanyperceiveditembiaseswereeliminated.Theitemreviewprocessincluded51reviewerswithanaverageof22yearsofexperienceineducation.TheORExtassessmentshavebeendeterminedtodemonstratestronglinkagetogradelevelacademiccontent,overall.Fulldocumentationoftheinitial2014linkagestudyandanew,independentalignmentstudyconductedinspring,2017isprovidedinAppendix3.1A.Noitemdevelopmentwasrequiredin2016-17.Thesummarysectionoftheindependentalignmentstudyreportstatesthat,“Oregon’sExtendedAssessments(ORExt)inEnglishLanguageArts,Mathematics,andSciencewereevaluatedinalow-complexityalignmentstudyconductedinSpringof2017.Averagesofreviewerprofessionaljudgmentsoverfiveseparateevaluationsweregathered,reviewed,andinterpretedinthepagesthatfollow.Inthethreeevaluationsthatinvolveddeterminingtherelationshipbetweenstandardsanditems,reviewersidentifiedsufficienttostrongrelationshipsamongassessmentcomponentsinallgradesandallsubjectareas.InthetwoevaluationsinvolvingAchievementLevelDescriptors,reviewersidentifiedthirtyinstancesofsufficienttostrongrelationshipsoutofthirty-fourpossiblerelationshipopportunitiesresultinginanoverallaffirmedrelationshipwithareasforrefinementsidentified.”BecausetheassessmentsdemonstratesufficienttostronglinkagetoOregon'sgeneraleducationcontentstandardsanddescriptivestatisticsdemonstratethateachcontentareaassessmentisfunctioningasintended,itisappropriatetodeducethatthesestandardsdefinetheexpectationsthatarebeingmeasuredbytheOregonExtendedassessments.TheOregonExtendedassessmentsyieldscoresthatreflectthefullrangeofachievementimpliedbyOregon'salternateachievementstandards.EvidenceofthisclaimisfoundinthestandardsettingdocumentationsubmittedinSection6.2.StandardsweresetforallsubjectareasonJune15-17,2015.Standardsincludedachievementleveldescriptorsandcutscores,whichdefineOregon'snewalternateachievementstandards(AAS).TheStateBoardofEducationofficiallyadoptedtheAASonJune25,2015.
3.1BAA-AAASLinkagetoGeneralContentStandardsCompleteresultsoftheanalysisofthelinkageofthenewEssentializedAssessmentFrameworks,(EAF),composedofEssentializedStandards(EsSt),togradelevelCCSSinEnglishlanguageartsandmathematicsandlinkedtoORSciandNGSSinscience,arepresentedinSection3.1A.TheclaimisthattheEsStaresufficientlylinkedtogradelevelstandards,whiletheORExtitemsarealignedtotheEsSt.Inadditiontopresentinglinkage
2016-2017 OR Extended Assessment Critical Element 3 – Page
49
informationbetweengradelevelcontentstandardsandtheEsSt,thelinkagestudypresentsalignmentinformationrelatedtotheitemsonthenewORExtincomparisontotheEsSt.Extendedassessmentshavebeendeterminedtolinksufficientlytogradelevelacademiccontentstandards.Fieldtestitemsareaddedeachyearbasedonitemalignmenttostandards.TheOregonExtendedassessmentslinktogradelevelacademiccontent,asreflectedintheitemdevelopmentprocess.OregonalsohadeachoperationalitemusedontheOregonExtendedassessmentevaluatedforalignmentaspartoftwocomprehensivelinkagestudies,oneperformedin2014andanindependentalignmentstudyperformedin2017(seeSection3.1A).Theprofessionalreviewersinaninternalstudyin2014andanindependentstudyinspring2017includedbothspecialandgeneraleducationexperts,withcontentknowledgeandexperienceinadditiontospecialeducationexpertise.Accordingtotheindependentlinkagestudyreport(seeAppendix3.1A),thespring2017reviewwasconductedbyexpertreviewerswithprofessionalbackgroundsineitherSpecialEducation(thepopulation),Assessment,orinOregon’sadoptedcontentstandards.Reviewerswereassignedtoreviewgrade-levelitemsrelativetotheirexperienceandexpertise.Inall,39reviewersparticipated.Thirty-four(34)participatedinall5evaluations:thirteen(13),fortheEnglishLanguageArtsreview,fifteen(15)fortheMathematicsreview,andsix(6)fortheSciencereview.AllparticipantswereassignedtoatleastonespecificcontentareaasshowninTable1.Note:Fourindividualswereassignedtotwoareasofreview.Thethirty-nineindividualswhoparticipatedinthestudyhadarobustlegacyofexperienceinthefieldandinthestate.Participantsrepresented25uniqueschooldistrictsacrossthestaterepresentingbothurbanandruralperspectives.All39oftheindividualsparticipatinginthestudyheldcurrentteachinglicenses.Twoindividualsalsoheldadministrativelicenses.Yearsofexperienceintheirarearangedfrom3–30yearsofexperiencewithanaverageof17yearsofexperience.(Mode=11years,Median=16years).Oneindividualindicated50yearsofexperienceinthefield.Threeofthe39individualsheldaBachelor’sdegreeonly.Thirty-sixheldaBachelor’sdegreeandatleastoneMaster’sdegree.TwoheldaBachelor’sdegree,atleastoneMaster’sdegree,andadoctoraldegree.Fourteen(36%)oftheindividualsidentifiedasexpertsinaspecificContentareaand25(64%)oftheindividualsidentifiedSpecialeducationastheirprimaryareaofexpertise.Theseskilledreviewersweretrainedbysynchronouswebinarsonlinkage/alignment,aswellasitemdepth,breadth,andcomplexityandthencompletedtheirratingsonlineviaBRT'sDistributedItemReview(DIR)websiteandonExcelspreadsheetssharedwiththeresearcherelectronically(seeAppendix3.1Bforanoverview).Mocklinkageratingswereconductedinordertoaddressquestionsandensureappropriatecalibration.Reviewersratedeachessentializedstandardona3-pointscale(0=nolink,1=sufficientlink,2=stronglink)asitrelatedtothestandardthetestdevelopershaddefinedforthatessentializedstandard.Itemswereevaluated,inturn,basedupontheiralignmenttotheessentializedstandardona3-pointscale(0=insufficientalignment,1=sufficientalignment,2=strongalignment).Whenaveragedacrossreviewers,1.00-1.29was
2016-2017 OR Extended Assessment Critical Element 3 – Page
50
consideredinthelowrange,1.30–1.69wassufficient,and1.70–2.0wasstrong.Additionalcommentwasrequestedforanyessentializedstandardoritemwhoselinkagewasrated0.Overall,the2017independentalignmentstudyconcludesthat:“First,reviewerswereaskedtoconductanaffirmationalreviewoftherationaleusedbytestdeveloperstoomitcertaincontentstandards.ThisfindingwasusedtoinferthatthefinalstandardsselectedforinclusionoromissioninOregon’sExtendedAssessmentwerechosenrationallyandthatthefinalscopeofcontentstandardscanbeconsideredjustifiableforthepopulationforthesubjectarea. Conclusion:Thisreview,withalowestaveragerateof.82(onascaleof1),permits theinference:thescopeofthestandardsselectedfortranslationtoEssentialized Standardswererationallyselected.Noneofthestandardsde-selected(for inaccessibilityorforbeingcoveredelsewhere)werestronglyidentifiedforre- inclusion,norwereidentifiedasacriticalholeforthispopulationofstudents.Second,reviewerswereaskedtoidentifythestrengthofthelinkbetweenthesourcestandardandtheEssentializedStandard.ThisfindingwasusedtoinferthattheprocessundertakentoessentializeagivenSourceStandarddidnotfundamentallyorcriticallyaltertheknowledgeorskillsetintendedbythesourcestandardforthispopulationofstudents(furtherconfirmingthatthecontentselectedforassessmentiscomparable). Conclusion:Thisreview,witharangeof1.5–1.9(onascaleof2)permitsthe inference:theEssentializedStandardswerefoundtolinksufficientlytothesource standardsonaveragebeyondthe“sufficient”averageof1.0.Third,reviewerswereaskedtoidentifythestrengthofthealignmentbetweentheEssentializedStandardsandtheitemsandtoreviewtheitemsdevelopedusingtheEssentializedStandardsforbias,andaccessibility.Thefindingfromthisreviewwasusedtoinferthattheitemswrittenforthisgradeandsubjectarea(usingtheseEssentializedStandards)wereadequatelylinkedtotheEssentializedStandards,werefreefrombias,andwereaccessibletostudentswithsignificantcognitivedisabilities. Conclusion:Thealignmentreview(1.32–1.89),accessibilityreview(.67**–1.0), andfreedomfrombiasreview(.65**–1.0)allpermittheinferencethatthetest itemsindicatearelationshipwiththesourcestandards,thetestitemsarenotoverly biasedtowardsoragainstanyparticulargroupofindividuals,andthetestitemsare writtensuchthatthecontentandintentcanbeaccessedbystudentswiththemost significantcognitivedisabilities.(**Note:thisrangewasskewedbyfeedbackfrom onereviewer--ELA-Grade3–whosecommentswerenotedinthisstudy.Removing thatindividual’scommentswouldresultinarangeof.90–1.0accessibilityrange and.89–1.0freedomfrombiasrangerespectively.)Fourth,reviewerswereaskedtoreviewthestatementsusedtodescribestudentachievementonthetest(theAchievementLevelDescriptors)andtheiralignmenttotheEssentializedStandardsthatthestudentsweretestedon.ThefindingfromthisreviewwasusedtoinferthattheskillsandachievementsdescribedbytheAchievementLevelDescriptorsforeachsubjectandgradelevelarealignedwiththecontentstandardbeingmeasured. Conclusion:Thereviewsrangingfrom.68*–1.0permittheinferencethatthe descriptionsmaderegardingstudentskillsetareanaccuratereflectionofthe
2016-2017 OR Extended Assessment Critical Element 3 – Page
51
standardsfromwhichtheassessmentwasdevelopedatallthreelevelsevaluated. (*OneoutlierforELA-Grade4providedareviewofa.52average).Fifth,andfinally,reviewerswereaskedtoreviewthealignmentoftheAchievementLevelDescriptorstotheitems.Thefindingfromthisreviewwasusedtoinferthateachiteminthedevelopedassessment(s)wasappropriatelyalignedtoitsassociatedAchievementLevelDescriptor(furtherconfirmingthatdecisionsmadeusingthistestwerealignedwiththeintentofthesourcestandard). Conclusion:Fourteenoftheseventeengrade-levelreviewsresultedinanaverage reviewerrangeof.67–1.0indicatinganappropriatealignmentbetweenALDsand theitemsaswritten.Thisreviewpermitstheinferencethat,overall,the AchievementLevelDescriptorsareaccuratereflectionsoftheitems.Inthree instances(Mathematics-Grades3and4,andELA-Grade8)theaveragealignmentby reviewerwas.5(indicatingthatoneofthetwoindividualsinthatcategorydidnot agreethattheitemsandALDswerealigned).” 3.2ValidityBasedonCognitiveProcessesEvidenceofcontentcoverageisconcernedwithjudgmentsabout“theextenttowhichthecontentdomainofatestrepresentsthedomaindefinedinthetestspecifications”(AERAetal.,2014,Standard4.12,p.89)7.Asawhole,theORExtiscomprisedofsetsofitemsthatsamplestudentperformanceontheintendeddomains.Theexpectationisthattheitemscoverthefullrangeofintendeddomains,withasufficientnumberofitemssothatscorescrediblyrepresentstudentknowledgeandskillsinthoseareas.Withoutasufficientnumberofitems,thepotentialexistsforavaliditythreatduetoconstructunder-representation(Messick,1989)6.TheORExtassessmentisbuiltuponavarietyofitemsthataddressawiderangeofperformanceexpectationsrootedintheCCSS,NGSS,andORScicontentstandards.ThechallengebuiltintothetestdesignisbasedfirstuponthecontentwithineachstandardinEnglishlanguagearts,mathematics,andscience.ThatcontentisRDBCinamannerthatisverifiedbyOregongeneralandspecialeducationteacherstodevelopassessmenttargetsthatareappropriateforstudentswiththemostsignificantcognitivedisabilities.Ourassessmentsutilizeuniversaldesignprinciplesinordertoincludeallstudentsintheassessmentprocess,whileeffectivelychallengingthehigherperformingstudents.ForstudentswhohaveverylimitedtonocommunicationandareunabletoaccesseventhemostaccessibleitemsontheORExt,anOregonObservationalRatingAssessment(ORora)wasfirstimplementedin2015-16.TheORoraiscompletedbyteachersanddocumentsthestudent'slevelofcommunicationcomplexity(expressiveandreceptive),aswellaslevelofindependenceinthedomainsofattention/jointattentionandmathematics.AcompletereportofORoraresultsfrom2016-17isprovidedinAppendix5.1D.Fifty-onereviewersanalyzedallORExtitemsforbias,sensitivity,accessibilitytothestudentpopulation,andalignmenttotheEssentializedStandards.Atotalof21reviewerswereinvolvedintheEnglishlanguageartsitemreviews.Anadditional21reviewerswere6Messick,S.(1989).Validity.InR.L.Linn(Ed.),Educationalmeasurement(3rded.,pp.13-103).NewYork:
AmericanCouncilonEducation.
2016-2017 OR Extended Assessment Critical Element 3 – Page
52
involvedintheMathematicsitemreviews.Scienceemployedninereviewers.Reviewerswereorganizedintogradelevelteamsoftwospecialeducatorsandonecontentspecialist.SubstantiveevidencethathasbeendocumentedsuggeststhattheORExtitemsaretappingtheintendedcognitiveprocessesandthattheitemsareattheappropriategradelevelthroughthelinkage/alignmentstudiesdocumentedabove,includingreviewsoflinkage,contentcoverage,anddepthofknowledge.AcomprehensivereportoftheitemreviewprocessisavailableinAppendix3.1A.3.3ValidityBasedonInternalStructure(ContentandFunction)TheOregonExtendedassessmentsreflectpatternsofemphasisthataresupportedbyOregoneducatorsasindicatedbythefollowingthreetablesthathighlightthebalanceofstandardrepresentationbygradelevelforEnglishlanguagearts,mathematics,andscienceontheORExt.Therepresentationratioscanbecalculatedbydividingthestandardsbythetotalwithineachrespectivecolumn.Forexample,inGrade3Reading,approximately25%oftheitemsareintheReadingStandardsforLiteraturedomain,asthatdomainhas4writtenEssentializedStandards(EsSt)outofthetotalof16(4/16=25%).ThetestblueprintsbelowdirectlycorrespondtothenumberofESwrittenineachdomainwithintheEssentializedAssessmentFrameworks(EAF)spreadsheets.ThereareadditionalgradelevelstandardsaddressedbytheEsSt,assomeEsStlinktomultiplegradelevelcontentstandards.However,theblueprintsbelowreflectonlythewrittenEsStandarethusanunderrepresentationofthebreadthofgradelevelcontentaddressedbytheORExt.EnglishLanguageArtsDomain Grade3 Grade4 Grade5 Grade6 Grade7 Grade8 Grade11RF 2 2 2 RI 4 4 4 5 5 5 5RL 4 4 4 5 5 5 5WR 4 4 4 4 4 4 4LA 2 2 2 2 2 2 2TOTAL 16 16 16 16 16 16 16Note.RF=ReadingStandards:FoundationalSkills.RI=ReadingStandardsforInformationalText.RL=ReadingStandardsforLiterature.WR=Writing.LA=Language.
2016-2017 OR Extended Assessment Critical Element 3 – Page
53
MathematicsDomain Grade3 Grade4 Grade5 Grade6 Grade7 Grade8 Grade11OAT 7 4 3 NBT 2 6 8 NOF 3 8 6 MED 8 5 4 GEO 2 3 2 3 3 4 7RPR 3 2 TNS 9 7 2 EXE 6 2 6 STP 5 6 3 5FUN 4 7NAQ 2ALG 2TOTAL 22 26 23 26 20 19 23Note.OAT=OperationsandAlgebraicThinking.NBT=NumbersandOperationsinBaseTen.NOF=NumbersandOperations–Fractions.MED=MeasurementandData.GEO=Geometry.RPR=RatioandProportionalRelationships.TNS=TheNumberSystem.EXE=ExpressionsandEquations.STP=StatisticsandProbability.FUN=Functions.NAQ=NumbersandQuantities.ALG=Algebra.Science
Domain Grade5 Grade8 Grade11LFS 4 9 8PHS 4 7 9ESS 4 6 6ETS 2 2 TOTAL 14 24 23Note.LFS=LifeScienceStandards.PHS=PhysicalSciences.ESS=EarthandSpaceSciences.ETS=Engineering,Technology,andApplications.TheprimarypurposeoftheORExtassessmentistoyieldtechnicallyadequateperformancedataongradelevelstatecontentstandardsforstudentswithsignificantcognitivedisabilitiesinEnglishlanguagearts,mathematics,andscienceatthetestlevel.Allscoringandreportingstructuresmirrorthisdesignandhavebeenshowntobereliablemeasuresatthetestlevel(seeSection4.1).Theprocessofaddressinganygapsorweaknessesinthesystemisaccomplishedviafield-testing(seeSection3.1A).
PointMeasureCorrelationsDistributionsofpointmeasurecorrelationsandoutfitmeansquarestatisticsforoperationalitemsareprovidedbelow,bycontentareaandgrade.Pointmeasure
2016-2017 OR Extended Assessment Critical Element 3 – Page
54
correlationsdisplayhowtheitemscorescorrelatewiththelatentoverallscore,whileoutfitmeansquarestatisticscloserto1.0denoteminimaldistortionofthemeasurementsystem.Allitemsincludedinthe2016-17operationalassessmentarerepresented.PointmeasurecorrelationsinELArangedfrom0.42to0.74.Alldatavisualizationswereconductedwithggplotinthetidyversepackage(Wickham,H.,2017).
English Language Arts − Grade 3
Point Measure Correlations
Item
Fre
quen
cies
0.4 0.5 0.6 0.7 0.8
02
46
8
English Language Arts − Grade 4
Point Measure Correlations
Item
Fre
quen
cies
0.4 0.5 0.6 0.7 0.80
24
68
English Language Arts − Grade 5
Point Measure Correlations
Item
Fre
quen
cies
0.4 0.5 0.6 0.7 0.8
02
46
8
English Language Arts − Grade 6
Point Measure Correlations
Item
Fre
quen
cies
0.4 0.5 0.6 0.7 0.8
02
46
8
English Language Arts − Grade 7
Point Measure Correlations
Item
Fre
quen
cies
0.4 0.5 0.6 0.7 0.8
02
46
8
English Language Arts − Grade 8
Point Measure Correlations
Item
Fre
quen
cies
0.4 0.5 0.6 0.7 0.8
02
46
8
2016-2017 OR Extended Assessment Critical Element 3 – Page
55
English Language Arts − Grade 11
Point Measure Correlations
Item
Fre
quen
cies
0.2 0.3 0.4 0.5 0.6 0.7 0.8
02
46
8
2016-2017 OR Extended Assessment Critical Element 3 – Page
56
Point-measurecorrelationsinmathematicsrangedfrom0.25to0.65.
Mathematics − Grade 3
Point Measure Correlations
Item
Fre
quen
cies
0.2 0.3 0.4 0.5 0.6 0.7 0.8
02
46
8
Mathematics − Grade 4
Point Measure Correlations
Item
Fre
quen
cies
0.2 0.3 0.4 0.5 0.6 0.7 0.8
02
46
8
Mathematics − Grade 5
Point Measure Correlations
Item
Fre
quen
cies
0.2 0.3 0.4 0.5 0.6 0.7 0.8
02
46
8
Mathematics − Grade 6
Point Measure Correlations
Item
Fre
quen
cies
0.2 0.3 0.4 0.5 0.6 0.7 0.8
02
46
8
Mathematics − Grade 7
Point Measure Correlations
Item
Fre
quen
cies
0.2 0.3 0.4 0.5 0.6 0.7 0.8
02
46
8
Mathematics − Grade 8
Point Measure Correlations
Item
Fre
quen
cies
0.2 0.3 0.4 0.5 0.6 0.7 0.8
02
46
8
2016-2017 OR Extended Assessment Critical Element 3 – Page
57
Point-measurecorrelationsinsciencerangedfrom0.47to0.72.
Mathematics − Grade 11
Point Measure Correlations
Item
Fre
quen
cies
0.2 0.3 0.4 0.5 0.6 0.7 0.8
02
46
8
Science − Grade 5
Point Measure Correlations
Item
Fre
quen
cies
0.3 0.4 0.5 0.6 0.7 0.8
02
46
8
2016-2017 OR Extended Assessment Critical Element 3 – Page
58
Science − Grade 8
Point Measure Correlations
Item
Fre
quen
cies
0.3 0.4 0.5 0.6 0.7 0.8
02
46
8
Science − Grade 11
Point Measure Correlations
Item
Fre
quen
cies
0.3 0.4 0.5 0.6 0.7 0.8
02
46
8
2016-2017 OR Extended Assessment Critical Element 3 – Page
59
OutfitMeanSquareDistributionsOutfitmeansquarevaluesbelow1.0demonstratethatvaluesaretoopredictableandperhapsredundant,whilevaluesabove1.0indicateunpredictability.Itemsabove2.0aredeemedinsufficientformeasurementpurposesandflaggedforreplacement.WhilemostOMSvaluesinELAwerebetween0.5and1.5,oneiteminGrade6wasabove2.0andwasremoved.
English Language Arts − Grade 3
Outfit Mean Squares
Item
Fre
quen
cies
0.0 0.5 1.0 1.5 2.0 2.5 3.0
01
23
45
6
English Language Arts − Grade 4
Outfit Mean Squares
Item
Fre
quen
cies
0.0 0.5 1.0 1.5 2.0 2.5 3.0
01
23
45
6
English Language Arts − Grade 5
Outfit Mean Squares
Item
Fre
quen
cies
0.0 0.5 1.0 1.5 2.0 2.5 3.0
01
23
45
6
English Language Arts − Grade 6
Outfit Mean Squares
Item
Fre
quen
cies
0.0 0.5 1.0 1.5 2.0 2.5 3.0
01
23
45
6
English Language Arts − Grade 7
Outfit Mean Squares
Item
Fre
quen
cies
0.0 0.5 1.0 1.5 2.0 2.5 3.0
01
23
45
6
English Language Arts − Grade 8
Outfit Mean Squares
Item
Fre
quen
cies
0.0 0.5 1.0 1.5 2.0 2.5 3.0
01
23
45
6
2016-2017 OR Extended Assessment Critical Element 3 – Page
60
English Language Arts − Grade 11
Outfit Mean Squares
Item
Fre
quen
cies
0.0 0.5 1.0 1.5 2.0 2.5 3.0
01
23
45
6
2016-2017 OR Extended Assessment Critical Element 3 – Page
61
WiththeexceptionofasingleiteminGrade7,whichwasreplaced,mathematicsOMSvaluesrangedfrom.35to1.7,demonstratingthattheitemsareperformingwithinexpectedranges.
Mathematics − Grade 3
Outfit Mean Squares
Item
Fre
quen
cies
0.0 0.5 1.0 1.5 2.0
01
23
45
6
Mathematics − Grade 4
Outfit Mean SquaresIte
m F
requ
enci
es
0.0 0.5 1.0 1.5 2.0
01
23
45
6Mathematics − Grade 5
Outfit Mean Squares
Item
Fre
quen
cies
0.0 0.5 1.0 1.5 2.0
01
23
45
6
Mathematics − Grade 6
Outfit Mean Squares
Item
Fre
quen
cies
0.0 0.5 1.0 1.5 2.0
01
23
45
6
Mathematics − Grade 7
Outfit Mean Squares
Item
Fre
quen
cies
0.0 0.5 1.0 1.5 2.0
01
23
45
6
Mathematics − Grade 8
Outfit Mean Squares
Item
Fre
quen
cies
0.0 0.5 1.0 1.5 2.0
01
23
45
6
2016-2017 OR Extended Assessment Critical Element 3 – Page
62
Withtwoexceptions,oneiteminGrade5ScienceandanotherinGrade8,ScienceOMSvaluesrangedfrom.47to1.75,demonstratingthattheitemsareperformingwithinexpectedrangeswithafewexceptionsthatwillberemoved.
Mathematics − Grade 11
Outfit Mean Squares
Item
Fre
quen
cies
0.0 0.5 1.0 1.5 2.0 2.5 3.0
01
23
45
6
Science − Grade 5
Outfit Mean Squares
Item
Fre
quen
cies
0.0 0.5 1.0 1.5 2.0 2.5
02
46
8
2016-2017 OR Extended Assessment Critical Element 3 – Page
63
Science − Grade 8
Outfit Mean Squares
Item
Fre
quen
cies
0.0 0.5 1.0 1.5 2.0 2.5
02
46
8
Science − Grade 11
Outfit Mean Squares
Item
Fre
quen
cies
0.0 0.5 1.0 1.5 2.0 2.5
02
46
8
2016-2017 OR Extended Assessment Critical Element 3 – Page
64
AnnualMeasureableObjectivesFrequencies&PercentagesAnnualMeasurableObjective(AMO)calculationswereconductedbaseduponstudentperformanceontheORExttiedtotheverticalscaleusingRaschmodeling.Overallresultsarelargelyconsistentwith2015-16,withapproximately50%ofstudentswithsignificantcognitivedisabilitiesachievingproficiencyacrossgradesandcontentareas.ELAresultsarepresentedinblue,mathematicsindarkgreen,andscienceinred.Thedatavisualizationspresentedbelowwereconductedwithggplotinthetidyversepackage(Wickham,H.,2017).
15.16%
38.22%
29.82%
16.80%
0
50
100
150
200
1 2 3 4AMO Performance Level
Frequency
Grade 3 ELA
21.78%
24.36%
26.76% 27.10%
0
50
100
150
1 2 3 4AMO Performance Level
Frequency
Grade 4 ELA
2016-2017 OR Extended Assessment Critical Element 3 – Page
65
25.04%
30.59%
23.70%
20.67%
0
50
100
150
1 2 3 4AMO Performance Level
Frequency
Grade 5 ELA
25.19% 25.00%25.95%
23.85%
0
50
100
1 2 3 4AMO Performance Level
Frequency
Grade 6 ELA
2016-2017 OR Extended Assessment Critical Element 3 – Page
66
26.08%
17.65%
28.24% 28.04%
0
50
100
150
1 2 3 4AMO Performance Level
Frequency
Grade 7 ELA
32.02%
18.97%
22.92%
26.09%
0
50
100
150
1 2 3 4AMO Performance Level
Frequency
Grade 8 ELA
2016-2017 OR Extended Assessment Critical Element 3 – Page
67
17.09%
24.10%
9.74%
49.06%
0
100
200
300
1 2 3 4AMO Performance Level
Frequency
Grade 11 ELA
30.28%
23.13%
38.27%
8.32%
0
50
100
150
200
1 2 3 4AMO Performance Level
Frequency
Grade 3 Math
2016-2017 OR Extended Assessment Critical Element 3 – Page
68
29.64%
35.60%
26.06%
8.69%
0
50
100
150
200
1 2 3 4AMO Performance Level
Frequency
Grade 4 Math
26.30%
34.17%33.17%
6.37%
0
50
100
150
200
1 2 3 4AMO Performance Level
Frequency
Grade 5 Math
2016-2017 OR Extended Assessment Critical Element 3 – Page
69
42.45%
12.05%
34.23%
11.28%
0
50
100
150
200
1 2 3 4AMO Performance Level
Frequency
Grade 6 Math
47.74%
3.54%
36.74%
11.98%
0
50
100
150
200
250
1 2 3 4AMO Performance Level
Frequency
Grade 7 Math
2016-2017 OR Extended Assessment Critical Element 3 – Page
70
39.72%
18.97%
37.15%
4.15%
0
50
100
150
200
1 2 3 4AMO Performance Level
Frequency
Grade 8 Math
34.36%
21.54%
35.38%
8.72%
0
50
100
150
200
1 2 3 4AMO Performance Level
Frequency
Grade 11 Math
2016-2017 OR Extended Assessment Critical Element 3 – Page
71
34.01%
15.15%
21.38%
29.46%
0
50
100
150
200
1 2 3 4AMO Performance Level
Frequency
Grade 5 Science
32.26%
18.64% 18.44%
30.66%
0
50
100
150
1 2 3 4AMO Performance Level
Frequency
Grade 8 Science
2016-2017 OR Extended Assessment Critical Element 3 – Page
72
Someconcernsarenotedinmathematics,whererelativelyhigherpercentagesofstudentsarescoringatLevel1andveryfewatLevel2.However,thisfindingisconsistentwiththerangeofpossiblescores,whereLevel2insomecasesonlyhastwopossiblescalescorepoints(e.g.,Grade7,whereLevel2existsbetween207-208scaledscores).Theadditionof1-2lowcomplexityitemsperassessmentwillbeeffectedinmathematicstoaddressthisconcern,aswell.3.4ValidityBasedonRelationstoOtherVariablesPerhapsthebestmodelforunderstandingcriterion-relatedevidencecomesfromCampbellandFiske(1959)7intheirdescriptionofthemulti-trait,multi-methodanalysis[wetranslatetheterm‘trait’tomean‘skill’].Inthisprocess(several)differenttraitsaremeasuredusing(several)differentmethodstoprovideacorrelationmatrixthatshouldreflectspecificpatternssupportiveoftheclaimbeingmade(thatis,providepositivevalidationevidence).Sometimes,thesevariousmeasuresareofthesameorsimilarskills,abilities,ortraits,andothertimestheyareofdifferentskills,abilities,ortraits.Wepresentdatathatquiteconsistentlyreflecthigherrelationsamongitemswithinanacademicsubjectthanbetweenacademicsubjects.Wealsopresentdatainwhichperformanceonitemsistotaledwithincategoriesofdisability,expectingrelationsthatwouldreflect
7Campbell,D.T.,&Fiske,D.W.(1959).Convergentanddiscriminantvalidationbythemulti-trait,multi-
methodmatrix.InW.A.Mehrens&R.L.Ebel(Eds.),Principlesofeducationalandpsychologicalmeasurement:Abookofselectedreadings(pp.273-302).Chicago,IL:RandMcNally&Company.
18.86%
16.09%
24.74%
40.31%
0
50
100
150
200
1 2 3 4AMO Performance Level
Frequency
Grade 11 Science
2016-2017 OR Extended Assessment Critical Element 3 – Page
73
appropriatedifferences(seeTindal,McDonald,Tedesco,Glasgow,Almond,Crawford,&Hollenbeck,2003).8
ConvergentandDivergentValidityDocumentationCriterionvalidityinformationisdifficulttodocumentwithAA-AAAS,asmostSWSCDdonotparticipateinanystandardizedassessmentoutsideoftheORExtand/orORorainOregon.DivergentvalidityevidenceisgarneredviacomparisonsofORExtresultstoORoraoutcomesshowsthatstudentswhoseORExtassessmentsarediscontinuedexhibitseriouslimitationsinattention,basicmathskills,andreceptiveandexpressivecommunicationskills.ThemedianORExtELAscoreforSWSCDwhoparticipatedintheORorawas4.0.ThemedianmathematicsORExtscorewas4.0,andthemedianscienceORExtscoreforSWSCDwhowereevaluatedwiththeORorawas0.0.PearsoncorrelationsbetweenthetotalrawscoresontheORExtandthetotalrawscoreontheORorawereconductedtoaddresstherelationshipbetweentotalperformanceoneachassessment.ThecorrelationbetweenELAandORorascoreswas0.56,betweenMathandORorascoreswas0.52,andbetweenScienceandORorascoreswas0.33.Asexpected,theORoraresultsprovidedivergentvalidityevidencefortheORExt.Wewouldnotexpectastrongrelationshipbetweenthescores,asstudentswhoseORExttestingisdiscontinuedaregenerallyunabletoaccesstheacademiccontentontheORExt,evenwiththerequisitereductionsindepth,breadth,andcomplexity.ConvergentevidencethattheORExtisassessingappropriateacademiccontentisprovidedbyQAandQTresponsestotheconsequentialvaliditysurvey.Respondentstothesurveygenerallyagreethat,“TheitemsintheOregonExtendedAssessmentaccuratelyreflecttheacademiccontent(whatthestudentshouldknow)thatmystudentswithsignificantcognitivedisabilitiesshouldbelearning,asdefinedbygradelevelcontentstandards(CCSS/NGSS)andtheEssentializedAssessmentFrameworks”(11%StronglyAgree&54%Agree).Inaddition,theyalsoagreedwiththestatementthat,“TheitemsintheOregonExtendedAssessment,whichprimarilyaskstudentstomatch,identify,orrecognizeacademiccontent,areappropriatebehaviorstoreviewtodeterminewhatmystudentswithsignificantcognitivedisabilitiesareabletodo”(18%StronglyAgree&64%Agree).TheconsequentialvalidityresultsdemonstratethattheORExtissamplingacademicdomainsthatthefieldofQAsandQTsdeemappropriateintheareaofacademics.
AnalysesWithinandAcrossSubjectAreasWeconductedcorrelationalanalysestofurtherexplorethevalidityoftheORExt.Wefirstdescribethepurposeoftheanalysis,aswellasouranticipatedresults.Wethendiscussourobservedresultsbeforeconcludingwithanoverallevaluativejudgmentofthevalidityofthetest.Inthecorrelationalanalysis,weexplorethecorrelationsamongstudents’totalscoresacrosssubjectareas.Thepurposeoftheanalysiswastoinvestigatehowstronglystudents’8Tindal,G.,McDonald,Tedesco,M.,Glasgow,A.,Almond,P.,Crawford,L.,&Hollenbeck,K.(2003).Alternate
assessmentsinreadingandmath:Developmentandvalidationforstudentswithsignificantdisabilities.ExceptionalChildren,69(4),481-494.
2016-2017 OR Extended Assessment Critical Element 3 – Page
74
scoresinoneareawererelatedtostudents’scoresinothersubjectareas.Ifthecorrelationswereexceedinglyhigh(e.g.,above.90),itwouldindicatethatthescoreastudentreceivesinanindividualsubjecthaslesstodowiththeintendedconstruct(i.e.,reading)thanwithfactorsidiosyncratictothestudent.Forexample,ifallsubjectareascorrelatedat.95,thenitwouldprovidestrongevidencethatthetestswouldbemeasuringaglobalstudent-specificconstruct(i.e.,intelligence),andnottheindividualsubjectconstructs.Wewouldexpect,however,thatthetestswouldcorrelatequitestronglygiventhatthesamestudentswereassessedmultipletimes.Therefore,wewouldexpectmoderatelystrongcorrelations(e.g.,0.7)simplybecauseofthewithin-subjectdesign.Idiosyncraticvarianceassociatedwiththeindividualstudentisthuscaptured.
CorrelationalAnalysesResultsFullresultsofthePearson’sproduct-momentcorrelationanalysisbycontentareaandgradelevelarereportedbelow.Theresultsaresignificant,yettheoverallcorrelationsacrosscontentareassuggestthatweareindeedmeasuringdifferent,thoughstronglyrelatedconstructs,withbetween-testscaledscorecorrelationsrangingfrom0.81to0.89.Grade3ContentAreaCorrelationsn-size ContentArea ELARITCorrelation MathRITCorrelation
601 ELA – .85*Math –
*p<.001Grade4ContentAreaCorrelationsn-size ContentArea ELARITCorrelation MathRITCorrelation
583 ELA – .84*Math –
*p<.001Grade5ContentAreaCorrelationsn-size ContentArea ELARIT
CorrelationMathRITCorrelation
ScienceRITCorrelation
588ELA – .85* .85*Math – .81*Science –
*p<.001Grade6ContentAreaCorrelationsn-size ContentArea ELARITCorrelation MathRITCorrelation
519 ELA – .83*Math –
*p<.001
2016-2017 OR Extended Assessment Critical Element 3 – Page
75
Grade7ContentAreaCorrelationsn-size ContentArea ELARITCorrelation MathRITCorrelation
508 ELA – .84*Math –
*p<.001Grade8ContentAreaCorrelationsn-size ContentArea ELARIT
CorrelationMathRITCorrelation
ScienceRITCorrelation
493ELA – .84* .89*Math – .83*Science –
*p<.001Grade11ContentAreaCorrelationsn-size ContentArea ELARIT
CorrelationMathRITCorrelation
ScienceRITCorrelation
572ELA – .83* .85*Math – .84*Science –
*p<.001ResultsofthePearson’sproduct-momentcorrelationanalysiswithinEnglishlanguagearts(ELA:Reading:Writing)arereportedbelowandsuggesthighcorrelationsbetweenELAandReading,asexpected,from.95to.97.WritingiscorrelatedwithELAfrom.91to.94andwithreadingfrom.79to.88.
2016-2017 OR Extended Assessment Critical Element 3 – Page
76
EnglishLanguageArtsSubscoreCorrelationsn-size GradeScaled
ScoreELARITCorrelation
ReadingRITCorrelation
WritingRITCorrelation
6013ELA – .95* .91*3Reading – .80*3Writing –
5834ELA – .95* .91*4Reading – .79*4Writing –
5885ELA – .96* .94*5Reading – .87*5Writing –
5196ELA – .97* .94*6Reading – .87*6Writing –
5087ELA – .97* .93*7Reading – .86*7Writing –
4938ELA – .97* .94*8Reading – .88*8Writing –
57211ELA – .97* .93*11Reading – .87*11Writing –
*p<.001TheORExtassessmentsappeartobemeasuringseparateconstructs,asintended,indicatedbythecorrelations.Nounexpectedandconsistenttestfunctioningstatisticsarepresentbasedonstudentcharacteristicsthatshouldnotberelated,suchasgenderandethnicity.Studentperformanceappearstobeprimarilyrelatedtoitemdifficultyandnottheresultofconstructirrelevantaspectsthathavebeenreviewed.
2016-2017 OR Extended Assessment Critical Element 4 – Page
77
CriticalElement4-TechnicalQuality:Other4.1ReliabilityTestreliabilitycanbeviewedthroughseverallenses,allofwhichdocumenthowconsistentlyanassessmentperformsacrossoccasions,contexts,andraters9.Typicalstrategiesforaddressingreliabilityincludedocumentationofinternalconsistency,split-halfreliability,andtest-retestreliability.Ifmultipleformsareimplemented,testformreliabilitydocumentationisalsorequisite.TheimplementationplanfortheORExtincludesinitialdocumentationofinternalconsistency(Cronbach'salpha).The2015-16technicalreportwillincludeinternalconsistencyestimates,split-halfreliabilityanalyses,aswellasasmalltest-retestassessmentofreliabilitycomparisonsbymeansofourpilottabletadministrationstudy.ThereisonlyonetestformfortheORExt,sotestformcomparisonsarenotpossible.
4.1ATestReliabilityMarginalreliabilityresults(truescorevariance/truescorevariance+errorvariance)demonstratethatthetestsarequitereliableatthetotaltestlevel.Fullreliabilitystatisticsforeachoftheoperationaltestsadministeredthisyearareprovidedbelow.Theseresultsdemonstratethatthetotaltestreliabilitieswerequitehigh,rangingfrom.87to.92.Eachtablebelowprovidesthecontentarea,grade,andthemarginalreliabilities.Alltestformswerecomposedof36operationaland12embeddedfield-testitems.EnglishLanguageArtsThetestreliabilitiesforELAwereinthehighrange,from.87to.92.
Grade MarginalReliability3 0.924 0.925 0.916 0.917 0.908 0.9111 0.87
MathematicsThetestreliabilitiesformathematicswereinthehighrange,from.88to.91.
Grade MarginalReliability3 0.914 0.915 0.906 0.90
9AmericanEducationalResearchAssociation(AERA),AmericanPsychologicalAssociation,&NationalCouncilonMeasurementinEducation(2014).Standardsforeducationalandpsychologicaltesting.Washington,DC:AERA
2016-2017 OR Extended Assessment Critical Element 4 – Page
78
Grade MarginalReliability7 0.918 0.8811 0.90
ScienceThetestreliabilitiesforsciencewereinthehighrange,from.87to.91.
Grade MarginalReliability5 0.918 0.8811 0.87
TestInformationFunctionsThetestinformationfunctionspublishedbelowalsoindicatethatthescalesexhibitareliabilitygreaterthanorequalto.80forallproficient-levelcutscores.EnglishlanguageartsTIFs
2016-2017 OR Extended Assessment Critical Element 4 – Page
79
2016-2017 OR Extended Assessment Critical Element 4 – Page
80
2016-2017 OR Extended Assessment Critical Element 4 – Page
81
2016-2017 OR Extended Assessment Critical Element 4 – Page
82
MathematicsTIFs
2016-2017 OR Extended Assessment Critical Element 4 – Page
83
2016-2017 OR Extended Assessment Critical Element 4 – Page
84
2016-2017 OR Extended Assessment Critical Element 4 – Page
85
ScienceTIFs
2016-2017 OR Extended Assessment Critical Element 4 – Page
86
2016-2017 OR Extended Assessment Critical Element 4 – Page
87
ValidationofORExtVerticalScalesThetestcharacteristiccurves(TCCs)forthegrade-levelassessmentsinELAandmathematicsdemonstrateincrementallyincreasinggrowthandtestdemandsacrossGrades3-8,withtheexceptionofGrade7mathematics.TheGrade7mathematicsassessmentwasrevisedtobemoredifficultlastyear,butclearlymoreelaborationofthiseffortisneededtoaddressitslocationontheTCC.Grade11andsciencetestsarenotverticallyscaled;TCCsarethusnotpresentedforGrade11orscience.AllRaschmodelscaling,aswellasthedatavisualizationsfortheTCCswereconductedintheRsoftware3.3.2environment(RCoreTeam,2016)usingther2Winstepspackage(Anderson,D.,2017).
-4 -2 0 2 4
010
2030
40
Θ
Exp
ecte
d To
tal R
aw S
core
Test Characteristic Curve
g3ELA.g4ELA.g5ELA.g6ELA.g7ELA.g8ELA.
2016-2017 OR Extended Assessment Critical Element 4 – Page
88
-4 -2 0 2 4
010
2030
40
Θ
Exp
ecte
d To
tal R
aw S
core
Test Characteristic Curve
g3Math.g4Math.g5Math.g6Math.g7Math.g8Math.
2016-2017 OR Extended Assessment Critical Element 4 – Page
89
4.1BOverallandConditionalStandardErrorsofMeasureTheaverageSEMassociatedwitheachcutscorefor2016-17studentdataarepresentedinthetablebelow,supportedbyaKEY.TheSEMsdecreasedinalmostallcasescomparedtolastyear,suggestingthatthemeasuresaremorereliablewhenstudenteligibilityismorestrictlycontrolled.SeeSection4.2belowformeansandstandarddeviationsbygradeandsubjectarea.SEM = StandardErrorofMeasureassociatedwiththecutscoretotheleft;averaged tothetenths'place.Level1 = DoesNotYetMeet(notincludedasthelowestlevelofproficiency)Level2 = NearlyMeetsLevel3 = MeetsLevel4 = ExceedsEnglishLanguageArtsGrade Level2 SEM Level3 SEM Level4 SEM AVG3 192 4.44 213 3.90 228 5.20 5.564 200 3.90 213 3.90 228 5.20 6.005 202 4.00 220 4.20 232 5.70 6.476 205 3.70 220 4.00 233 5.70 6.417 208 3.70 222 4.00 236 5.70 6.538 213 3.60 224 3.90 236 5.40 6.4611 899 3.80 920 4.30 927 5.40 8.41
MathematicsGrade Level2 SEM Level3 SEM Level4 SEM AVG3 192 3.90 201 3.80 218 5.10 5.104 193 3.80 206 3.80 219 5.00 4.965 193 4.20 206 3.80 220 4.40 5.306 204 3.60 208 3.60 222 4.50 4.957 207 3.80 209 3.90 223 5.30 5.338 208 3.70 212 3.70 226 4.40 4.9511 901 3.60 907 3.60 922 4.85 4.96
ScienceGrade Level2 SEM Level3 SEM Level4 SEM AVG5 506 3.70 517 4.10 530 5.60 6.648 810 3.80 820 4.40 831 6.10 7.3611 901 3.60 914 4.00 929 6.20 7.88
4.1CClassificationAccuracy&Consistency
Resultsfromthe2016-17ORExttestadministrationwereanalyzedusingRudner’sclassificationindex(Rudner,2005).Resultscloserto1.0indicatethelikelihoodthatastudentwasappropriatelyclassifiedasproficientornotproficient(accuracy)andthelikelihoodthatthestudentwouldbeclassifiedinthesamecategorygivenanadditionaltest
2016-2017 OR Extended Assessment Critical Element 4 – Page
90
administration.Thecalculationutilizesitemdifficultyandthetavaluedistributions,aswellasrelatedstandarderrorsofmeasurement,togenerateprobabilisticestimatesbasedononetestadministration.Completeresults,generatedfromthecacIRTpackageinR,areprovidedbelow.Resultsdenoteveryhighlevelsofclassificationaccuracyandconsistency.TestClassificationAccuracyGrade EnglishLanguage
ArtsMathematics Science
3 0.94 0.92 -4 0.94 0.92 -5 0.95 0.91 0.956 0.95 0.92 -7 0.95 0.93 -8 0.94 0.88 0.9411 0.95 0.91 0.93TestClassificationConsistencyGrade EnglishLanguage
ArtsMathematics Science
3 0.91 0.89 -4 0.92 0.88 -5 0.92 0.87 0.926 0.93 0.88 -7 0.92 0.90 -8 0.92 0.84 0.9211 0.93 0.88 0.93TheORExtisnotacomputer-adaptiveinstrumentsoestimateprecisiondocumentationbaseduponthattestdesignisnotprovided.4.2FairnessandAccessibilityThestatehastakenstepstoensurefairnessinthedevelopmentoftheassessments,includingananalysisofeachtestitembyOregonteachersnotonlyforlinkagetostandards,butalsoforaccess,sensitivity,andbias(seeAppendix3.1A).Inaddition,wereviewedtestfunctioningasrelevanttorace/ethnicityanddisabilitysubgroups.Thisprocessincreasesthelikelihoodthatstudentsarereceivinginstructioninareasreflectedintheassessment,andalsothattheitemsarenotbiasedtowardaparticulardemographicorsub-group.
DifferentialItemFunctioningAnalysesToinvestigateDifferentialItemFunctioning(DIF),theMantel-Haenszeltestusinga
purificationprocesswasconducted(Holland&Thayer,1988;Kamata&Vaughn,2004)withtheRsoftwareusingthedifRpackage(Magisetal.,2013).WhenusingtheMantel-HaenszeltesttoinvestigateDIF,contingencytablesareconstructed,andtheresultingoddsforthefocalgroupansweringtheitemcorrectlyarecomparedtotheoddsforthereferencegroup.Givenn-sizelimitations(Scott,etal.,2009),wewereabletoconducttwoanalyses:
2016-2017 OR Extended Assessment Critical Element 4 – Page
91
a)White/Non-Whiteandb)Male/Female.WhitesandMaleswerethefocalgroupsandNon-WhitesandFemaleswerethereferencegroups,respectively.Thecontingencytablesummarizescorrectandincorrectresponsestoeachitembyrespondents’totalrawscorebysubgroup(Kamata&Vaughn,2004).Ifthereisnodifferenceinperformanceforthetwogroups,theoddsratioofthefocalgroupperformancetoreferencegroupperformancewillequalone.Anoddsratiogreaterthanonemeansthefocalgroupisperformingbetterthanthereferencegroup,withtheoppositebeingtrueforoddsratioslessthanone.
ThedifRpackagecontainsabuiltinalgorithmtoconductpurificationautomatically,sowewereinterestedinhowthisalgorithmfunctionedrelativetotheiterationsconductedmanuallyusingSPSS.WeusedcriteriaoutlinedbytheEducationalTestingService(ETS)forDIFClassification(Holland&Thayer,1988)todeterminewhetherornotitemsexhibitedDIF,asthedifRpackagereportsdeltavaluesbydefault,definedas
∆!"= −2.35 ∗ 𝑙𝑛(𝛼!!) (1)where𝑙𝑛(𝛼!!) = 𝛽!"! .TheHollandandThayercriteriafor∆!" isasfollows:
∆!" ≤ 1.0=“A”items,1 ≤ ∆!" ≤ 1.5=“B”items,and ∆!" ≥ 1.5=“C”items.
TheHollandandThayercriteriawereusedforallMantel-Haenszelanalyses.Itemsthatwereflaggedas“C”levelitemswerereviewedbyBRTresearchersforpotentialbiases.Ifbiasesareidentified,theitemisremovedfromtheitempool.DIFanalyseswereperformedexpostfactoonthe2015-16ORExtoperationalitemstoaddresslongitudinaltrends.OnlythreeELAitemswereidentifiedasexhibitinga“C”levelDIFacrossboth2016and2017.ThosethreeELAitems,oneinGrade5thatexhibitedDIFthatprivilegedWhiteexaminees,oneinGrade4thatprivilegedFemaleexaminees,andoneinGrade8thatprivilegedFemaleexaminees,wereremovedandwillnotbeusedin2017-18orthereafter.DIFanalyseswillalsobeperformedinthe2017-18schoolyeartocontinuetoaddressDIFlongitudinally.Allitems,includingfieldtestitems,wereincludedintheanalyses.Thereareatotalof48itemsoneachassessment.
2016-2017 OR Extended Assessment Critical Element 4 – Page
92
WithintheWhite/Non-Whiteanalysis,10outof18itemsflaggedas“C”levelitemsprivilegedNon-WhitetestparticipantsinELA,2outof5privilegedNon-WhitetestparticipantsinMathematics,and2outof7privilegedNon-WhitetestparticipantsinScience.Overall,DIFflaggingbasesonracewasrelativelybalanced,with14privilegingstudentswhowereNon-Whiteand16privilegingstudentswhowereWhite.White/Non-WhiteDIFAnalysesResultsContentArea
GradeNon-Whiten
Whiten
“C”ItemFrequency
ItemsRemovedCount
EnglishLanguageArts
3 248 359 3 04 263 320 2 05 247 348 1 16 232 292 7 07 208 302 3 08 202 304 0 N/A11 222 363 2 0
Mathematics 3 247 354 1 04 264 323 0 N/A5 248 349 0 N/A6 232 291 2 07 208 301 1 08 201 305 0 N/A11 222 363 1 0
Science 5 250 344 2 08 199 300 5 011 219 359 0 N/A
2016-2017 OR Extended Assessment Critical Element 4 – Page
93
IntermsoftheMale/Femaleanalyses,10outof16itemsflaggedas“C”levelitemsprivilegedFemalesinELA,4outof9flaggeditemsprivilegedFemalesinMathematics,and8outof11flaggeditemsprivilegedFemalesinScience.Overall,DIFflaggingbasedonsexwasrelativelybalanced,with22privilegingFemalesand14privilegingMales.Male/FemaleDIFAnalysesResultsContentArea
Grade Femalen
Malen
“C”ItemFrequency
ItemsRemovedCount
EnglishLanguageArts
3 181 426 4 04 200 383 2 15 209 386 2 06 171 353 1 07 152 358 1 08 162 344 4 111 220 365 2 0
Mathematics 3 180 421 1 04 201 386 1 05 208 389 1 06 170 353 0 N/A7 152 357 4 08 162 344 1 011 221 364 1 0
Science 5 209 385 4 08 162 337 5 011 216 362 2 0
Race–EthnicityPercentagesandTotalsbyContentAreaandGradeLevel
ThefullethnicanddisabilitydemographicsforstudentstakingtheORExtarereportedbelow.Studentsethnicity/racewasreportedinsevencategories:(a)AmericanIndian/AlaskanNative,(b)Asian,(c)BlackorAfrican-American,(d)Multi-ethnic,(e)NativeHawaiianorOtherPacificIslander,(f)Hispanic,or(g)White.ThemajorityofstudentswerereportedasWhite(55-62%)orHispanic(22-29%).Theseresultsarelargelyconsistentwiththedemographicsreportedforthegeneralassessments,thoughpercentagestakingtheORExtareslightlyhigherformoststudentsofcolorandgenerallylowerforstudentswhoareAsianorWhite(seeAppendix4.2).
2016-2017 OR Extended Assessment Critical Element 4 – Page
94
Englishlanguagearts GradeEthnicity-Race 3 4 5 6 7 8 11
AmericanIndian/AlaskaNative
.02 .02 .02 .03 .03 .02 .03
Asian .02 .04 .03 .03 .05 .04 .02BlackorAfrican-American
.03 .03 .02 .03 .03 .03 .05
Multi-ethnic .05 .07 .06 .08 .05 .06 .06NativeHawaiianorOtherPacificIslander
* * * * * * *
Hispanic .27 .29 .26 .27 .24 .23 .22White .59 .55 .59 .56 .60 .60 .62Totaln-sizes 584 571 574 507 504 490 563Note.*n<10;percentagesmaynotaddto1.0duetorounding.Mathematics GradeEthnicity-Race 3 4 5 6 7 8 11
AmericanIndian/AlaskaNative
.02 .02 .02 .03 .03 .02 .02
Asian .02 .04 .03 .03 .05 .04 .02BlackorAfrican-American
.03 .03 .02 .03 .03 .03 .05
Multi-ethnic .05 .07 .06 .08 .05 .06 .06NativeHawaiianorOtherPacificIslander
* * * * * * *
Hispanic .27 .29 .26 .27 .24 .23 .22White .59 .55 .59 .56 .60 .60 .62Totaln-sizes 578 575 575 506 502 490 563Note.*n<10;percentagesmaynotaddto1.0duetorounding.
2016-2017 OR Extended Assessment Critical Element 4 – Page
95
Science GradeEthnicity-Race 5 8 11AmericanIndian/AlaskaNative
.02 .02 .03
Asian .03 .04 .02BlackorAfrican-American .03 .03 .05Multi-ethnic .06 .06 .06NativeHawaiianorOtherPacificIslander
* * *
Hispanic .27 .23 .22White .59 .60 .62Totaln-sizes 573 483 557Note.*n<10;percentagesmaynotaddto1.0duetorounding. StudentreportedexceptionalitiesincludedIntellectualDisability(ID),HearingImpairment(HI),VisualImpairment(VI),Deaf-Blindness(DB),CommunicationDisorder(CD),EmotionalDisturbance(ED),OrthopedicImpairment(OI),TraumaticBrainInjury(TBI),OtherHealthImpairment(OHI),AutismSpectrumDisorder(ASD),andSpecificLearningDisability(SLD).ThemajorityofstudentswhoparticipatedintheORExtwerestudentswithID(30-45%)andstudentswithASD(28-34%),followedbystudentswithOHI(11-16%).ODEpolicyfor2015-16changedtorequirestudentswhoparticipateintheORExttotaketheassessmentinallrelevantcontentareas.Thereisthusverylittlechangeintermsofparticipationpercentagesacrosscontentareas,asevidencedbythetotaln-sizespergradeleveldisplayedbelow.
ExceptionalityPercentagesByContentAreaandGradeLevelEnglishlanguagearts GradeCategory 3 4 5 6 7 8 11ID10 .30 .34 .37 .36 .39 .45 .43HI20 * * * * * * *VI40 * * * * * * *DB43 * * * * * * *CD50 .09 .08 .07 .06 .05 .03 .04ED60 .02 .02 .02 * .02 * .02OI70 .03 .03 .04 .03 .05 .03 .03TBI74 * * * * * * *OHI80 .14 .14 .16 .13 .12 .11 .11ASD82 .29 .29 .28 .34 .30 .31 .28SLD90 .06 .06 .05 .05 .06 .03 .06Totaln-sizes 584 571 574 507 504 490 563Note.*n<10;percentagesmaynotaddto1.0duetorounding.
2016-2017 OR Extended Assessment Critical Element 4 – Page
96
Mathematics GradeCategory 3 4 5 6 7 8 11ID10 .30 .34 .37 .36 .39 .45 .43HI20 * * * * * * *VI40 * * * * * * *DB43 * * * * * * *CD50 .09 .08 .07 .06 .05 .03 .05ED60 .02 .02 .02 * .02 * .02OI70 .04 .03 .04 .03 .05 .03 .03TBI74 * * * * * * *OHI80 .16 .14 .16 .14 .12 .11 .11ASD82 .31 .29 .29 .34 .30 .31 .28SLD90 .06 .06 .05 .05 .06 .03 .05Totaln-sizes 578 575 575 506 502 490 563Note.*n<10;percentagesmaynotaddto1.0duetorounding.Science GradeCategory 5 8 11ID10 .36 .45 .42HI20 * * *VI40 * * *DB43 * * *CD50 .06 .03 .05ED60 .02 * .03OI70 .04 .04 .03TBI74 * * *OHI80 .16 .11 .11ASD82 .29 .31 .28SLD90 .05 .03 .05Totaln-sizes 573 483 557Note.*n<10;percentagesmaynotaddto1.0duetorounding.
ObservedMeansandStandardDeviationsThefollowingtablesprovideinformationregardingobservedmeansandstandarddeviationsbycontentareaandgradelevel.TheGrade3-8Englishlanguageartsandmathematicsscaledscoresarecenteredon200,whileallGrade11scoresarecenteredon900(toreinforcethattheyarenotontheverticalscale).Scienceiscenteredon500atGrade5andcenteredon800atGrade8.Theverticallyscaledscoresgenerallyconveyincrementalgainsinachievementacrossgradelevels,thoughtheresultssuggestsmalllossesappearingatGrade8inELA.Thesescaleswereselectedtoclearlydeterminewhetherscoresareonthesamescaleandalsotodifferentiateamongthestatewide
2016-2017 OR Extended Assessment Critical Element 4 – Page
97
assessmentsinusetoavoidconfusion(i.e.,SBA,OAKS,ORExt,ELPA,KA).ThegeneralpatternisthatRITscoresdecreasedfrom2014-15to2015-16.Thisdecreaseisattributednottothescale,nortodecelerationofgrowth,buttothesubstantiveshiftinthetestedstudentpopulationasaresultofODEeligibilityguidelines.Thescalefrom2015-16to2016-17appearstohavestabilizedbecausethestudentpopulationtestedwasmoreconsistent.2014-15RITScores
ELA Math Science
Grade Mean SD Mean SD Mean SD3 219.3 24.6 201.5 20.8
4 222.8 23.6 204.8 19.8 5 224.9 25.0 205.3 18.1 517.6 25.6
6 226.3 24.0 207.7 17.7 7 226.4 25.0 207.9 19.0 8 225.4 24.1 207.8 17.3 822.1 25.8
11 922.5 28.5 903.8 21.1 920.8 27.7
2015-16RITScores
ELA Math Science
Grade Mean SD Mean SD Mean SD3 210.3 23.0 197.6 20.2
4 212.3 22.9 198.1 18.7 5 217.1 24.5 201.2 17.2 514.2 22.1
6 220.1 25.5 204.8 17.6 7 223.6 28.9 205.4 19.0 8 221.2 24.8 206.7 17.2 819.0 25.6
11 920.7 27.7 902.3 20.0 918.0 24.92016-17RITScores
ELA Math Science
Grade Mean SD Mean SD Mean SD3 209.64 21.73 196.16 18.96 - -4 213.13 23.38 198.45 17.98 - -5 213.85 25.01 198.37 19.54 513.65 24.596 216.65 23.76 203.29 17.43 - -7 220.53 23.88 205.13 19.87 - -8 219.48 24.28 205.92 16.26 817.96 24.3611 922.05 26.37 903.07 17.57 919.41 24.25
2016-2017 OR Extended Assessment Critical Element 4 – Page
98
ObservedMeansReportedbySexThefollowingtablesprovideinformationregardingaveragestudentperformancebygradelevelandsex(Female/Male)ineachofthecontentareasassessedontheORExt.SignificantdifferencesbasedonaWelchtwosamplet-testarenotedinGrade4ELA,Grades4,5,and8inmathematicsandGrades5and8inscience.EnglishLanguageArts GradeSex 3 4 5 6 7 8 11Female 209.65 210.30* 211.85 216.44 219.46 218.88 921.56Male 209.56 214.71* 214.70 217.55 220.99 220.33 922.39Note.*p<.05Mathematics GradeSex 3 4 5 6 7 8 11Female 193.31 194.16* 195.27* 201.91 202.63 203.67* 901.87Male 196.73 200.76* 199.87* 204.60 206.41 207.31* 904.05Note.*p<.05Science GradeSex 5 8 11Female 510.36* 814.84* 916.67Male 515.00* 820.02* 920.88Note.*p<.05
ObservedMeansReportedbyRaceThefollowingtablesprovideinformationregardingaveragestudentperformancebygradelevelandrace(Non-White/White)ineachofthecontentareasassessedontheORExt.Significantdifferencesarenotedbytwosamplet-testsinELAGrade3and8andinGrade8inScience.EnglishLanguageArts GradeRace 3 4 5 6 7 8 11Non-White 207.45* 212.28 212.65 217.34 220.35 216.95* 921.01White 211.06* 213.91 214.41 217.07 220.67 221.81* 922.73Note.*p<.05
2016-2017 OR Extended Assessment Critical Element 4 – Page
99
Mathematics GradeRace 3 4 5 6 7 8 11Non-White 194.57 197.63 197.71 203.73 205.18 205.82 903.24White 197.06 199.15 198.62 203.73 205.37 206.40 903.24Note.*p<.05Science GradeRace 5 8 11Non-White 511.42 815.74* 917.79White 514.71 820.10* 920.86Note.*p<.05
ObservedMeansReportedbyExceptionalityStatusThefollowingtablesprovideinformationregardingaveragestudentperformancebygradelevelandexceptionalitycategoryineachofthecontentareasassessedontheORExt.StudentswithSLDweregenerallythehighestperforminggroup,thoughstudentswithCDandEDperformedhigheratcertaingradelevels/contentareas.ThelowestperforminggroupwasconsistentlystudentswithOI,followedbystudentswithIDorASD,dependingupongradelevel.EnglishLanguageArtsExceptionality GradeCode 3 4 5 6 7 8 11ID10 209.12 210.70 209.76 214.33 219.82 219.62 916.96HI20 * * * * * * *VI40 * * * * * * *DB43 * * * * * * *CD50 216.82 222.02 229.63 230.13 234.08 232.88 939.00ED60 226.38 222.21 235.29 236.17 229.80 * 941.36OI70 185.50 181.22 186.52 196.93 187.09 198.24 911.32TBI74 * * * * * * *OHI80 212.31 218.47 213.51 220.28 221.47 226.15 935.03ASD82 206.11 210.82 214.81 214.46 219.90 216.66 918.10SLD90 228.34 228.60 238.22 237.22 236.03 241.06 942.35Note.*n<10;theGrade11scaleisuniqueinELAandnottobecomparedtoGrades3-8.
2016-2017 OR Extended Assessment Critical Element 4 – Page
100
MathematicsExceptionality GradeCode 3 4 5 6 7 8 11ID10 194.91 196.47 195.61 201.23 204.41 205.61 900.20HI20 * * * * * * *VI40 * * * * * * *DB43 * * * * * * *CD50 201.28 207.77 211.11 213.63 218.92 214.88 915.00ED60 213.15 210.43 214.36 212.00 211.20 * 916.07OI70 176.30 173.11 178.10 188.50 178.22 192.06 894.89TBI74 * * * * * * *OHI80 197.45 202.46 198.53 204.93 208.92 209.85 910.87ASD82 193.71 194.81 198.68 203.02 203.37 204.30 901.01SLD90 214.74 215.54 214.00 218.52 219.71 219.06 915.73Note.*n<10;theGrade11scaleisuniqueinMathandnottobecomparedtoGrades3-8.ScienceExceptionality GradeCode 5 8 11ID10 509.99 818.50 915.83HI20 * * *VI40 * * *DB43 * * *CD50 529.68 828.00 936.50ED60 539.79 * 943.79OI70 489.43 798.12 908.89TBI74 * * *OHI80 517.01 826.71 932.69ASD82 510.60 813.41 914.13SLD90 539.46 839.00 938.00Note.*n<10;allGradelevelscalesareuniqueinScienceandnottobecompared.
GraphsofObservedMeansByDisabilityThegraphsbelowconveyinformationsimilartothatsharedaboveingraphicform.Thegraphicsinclude95%confidenceintervalerrorbars,sodeterminingwhichsubgroupsperformedinamannerthatissignificantlybetterthanothersisreadilyapparentbylookingatthelocationoftheerrorbars.Errorbarsthatdonotoverlapintermsofthey-scalearesignificantlydifferent.Onlystudentswhogenerallyhadmorethan10membersateachgradelevelarereported.ThisrequiredtheremovalofgraphsforstudentsintheHI,VI,DB,andTBIcategories.
2016-2017 OR Extended Assessment Critical Element 4 – Page
101
StudentswithOIareagainthelowestperforminggroup,beingsignificantlyoutperformedbyallothersubgroups.StudentswithSLDareconsistentlyoutperformingmostpeers,withstudentswithEDandCDperformingatsimilarlyhighlevels.StudentswithOIareconsistentlythelowestperforminggroup,whichledtoconcernsregardingtestaccessibility.However,theresultsoflastyear’sconsequentialvaliditystudydemonstratedthattheOIlabelisinsufficienttofullydescribetheseverityandrangeofconcomitantdisabilitiesthatstudentswhoseprimarylabelisOIconveys.
180
200
220
240
3 4 5 6 7 8Grade
ELA
Mea
n
ExceptionalityID
CD
ED
OI
OHI
ASD
SLD
Average ELA RIT Scores By Exceptionality
2016-2017 OR Extended Assessment Critical Element 4 – Page
102
900
910
920
930
940
950
Grade 11
ELA
Mea
n
ExceptionalityID
CD
ED
OI
OHI
ASD
SLD
Average ELA RIT Scores By Exceptionality
2016-2017 OR Extended Assessment Critical Element 4 – Page
103
160
180
200
220
3 4 5 6 7 8Grade
Mat
h M
ean
ExceptionalityID
CD
ED
OI
OHI
ASD
SLD
Average Math RIT Scores By Exceptionality
2016-2017 OR Extended Assessment Critical Element 4 – Page
104
890
900
910
920
Grade 11
Mat
h M
ean
ExceptionalityID
CD
ED
OI
OHI
ASD
SLD
Average Math RIT Scores By Exceptionality
2016-2017 OR Extended Assessment Critical Element 4 – Page
105
490
510
530
Grade 5
Sci
ence
Mea
n
ExceptionalityID
CD
ED
OI
OHI
ASD
SLD
Average Science RIT Scores By Exceptionality
2016-2017 OR Extended Assessment Critical Element 4 – Page
106
780
800
820
840
Grade 8
Sci
ence
Mea
n
ExceptionalityID
CD
ED
OI
OHI
ASD
SLD
Average Science RIT Scores By Exceptionality
2016-2017 OR Extended Assessment Critical Element 4 – Page
107
Grades3-8ObservedandUnconditionalGrowthExpectationsTheORExtwasredesignedin2014-15tosupportgrowthdeterminationsinGrades3-8inEnglishlanguageartsandmathematics.Averticalscaleusingabalanceddesignwasusedtodeveloptheinitialscale.Nowthatweareinthethirdyearofadministration,itbecamepossibletomodelgrowthexpectationsforELAandMathforSWSCDwhotooktheORExt.ThefollowinggraphsconveytheaveragegrowthexpectationsforSWSCDinOregonandshouldprovidesomecontextforunderstandingtypicalperformanceandaveragegrowthinIndividualizedEducationProgram(IEP)meetings.TheODEchangedtheeligibilitycriteriaforSWSCDtoparticipateintheORExtinthe2015-16schoolyear.Thishadanimpactonthetestedpopulation,astheexpectationsweremore
900
910
920
930
940
950
Grade 11
Sci
ence
Mea
n
ExceptionalityID
CD
ED
OI
OHI
ASD
SLD
Average Science RIT Scores By Exceptionality
2016-2017 OR Extended Assessment Critical Element 4 – Page
108
prescriptive,andstudentpopulationsdecreasedbyanaverageof40%ineachcontentareaandgradeleveltested.ThischangealsoaffectedORExttestresults,asthestudentswhoparticipatedinthefirstadministrationbutnotinsubsequentadministrationsweregenerallyveryhighachieving.TogenerategrowthestimatesthatmatchedtheintendedstudentpopulationfortheORExt,namelystudentswhodidnotexittheassessmentafterthe2015administration,alldatasetsforgrowthmodelingexcludedthegroupofstudentswhoparticipatedinonlythe2015administration.Studentswhosegradeleveladvancementwasnottypicalwerealsoexcluded(n=18exclusionsinELAandmath,respectively).Allotherparticipantsweremaintained.Theobservedcohortmeansarerepresentedbelowforcomparisonpurposes.InELA,thescoresatGrade3averageaRITscoreof205.72.ByGrade8,theaverageRITscoreinELAis218.99.Intermsofobservedmeans,studentsthusgrowatotalof13.27RITscorepointsfromGrades3to8inELA,foranaverageannualgrowthrateof2.21RITscorepointsperyear.Inmathematics,theaverageGrade3RITscorewas193.20.ByGrade8,theaveragescorewas205.78.Students’observedmeansthusincreasedby12.58RITscorepoints,foranaverageannualgrowthrateof2.10RITscorepointsperyear.EnglishLanguageArtsObservedMeans2015–2017byCohort
Cohort1 Cohort2 Cohort3 Cohort4Grades 3-4-5 4-5-6 5-6-7 6-7-8
3 205.72 4 209.74 210.37 5 211.40 213.95 214.71 6 214.44 217.03 215.067 218.97 218.668 218.99
MathematicsObservedMeans2015-2017byCohort
Cohort1 Cohort2 Cohort3 Cohort4Grades 3-4-5 4-5-6 5-6-7 6-7-8
3 193.20 4 196.22 196.22 5 196.74 199.47 200.43 6 202.18 203.05 200.707 204.21 202.718 205.78
Observedmeanshideasubstantialamountofinformation,however,astheydonotaccountforthevarianceinscoresthatexistsinthepopulation.Wethusconductedunconditionalgrowthmodelstoparseoutthevarianceassociatedwitheachinterceptandslopeestimate.Weincludedmultiplecohortstoaddresstheobservednon-linearityinthegrowthestimates.AlldatapreparationandanalyseswereconductedintheRsoftware3.3.2environment(RCoreTeam,2016)usingthelme4package(Bates,Maechler,Bolker,&Walker,2015).Inaddition,thedatavisualizationsbelowwereconductedwithggplotinthe
2016-2017 OR Extended Assessment Critical Element 4 – Page
109
tidyversepackage(Wickham,H.,2017).Cohorteffectswereaddressedbyaveragingacrossoverlappinggrades;however,theprocessofaveragingovercohortsshouldcontinueannually.UnconditionalModel-PredictedELAMeans2015–2017byCohort
Cohort1 Cohort2 Cohort3 Cohort4 Grades 3-4-5 4-5-6 5-6-7 6-7-8 AVG
SlopeEstimate 2.73 1.93 1.93 1.81 2.103 206.21 206.214 208.94 210.98 209.965 211.67 212.91 214.95 213.186 214.84 216.88 215.76 215.837 218.81 217.57 218.198 219.38 219.38
3 4 5 6 7 8
200
205
210
215
220
English Language Arts Growth
Grade
RIT
Sco
re
Cohort 1Cohort 2Cohort 3Cohort 4Average
2016-2017 OR Extended Assessment Critical Element 4 – Page
110
UnconditionalModel-PredictedMathematicsMeans2015–2017byCohort Cohort1 Cohort2 Cohort3 Cohort4
Grades 3-4-5 4-5-6 5-6-7 6-7-8 AVGSlopeEstimate 1.66 2.85 1.78 2.47 2.19
3 193.72 193.724 195.38 196.42 195.905 197.04 199.27 200.77 199.036 202.12 202.55 200.57 201.757 204.33 203.04 203.698 205.51 205.51
3 4 5 6 7 8
190
195
200
205
210
Mathematics Growth
Grade
RIT
Sco
re
Cohort 1Cohort 2Cohort 3Cohort 4Average
2016-2017 OR Extended Assessment Critical Element 4 – Page
111
Theunconditionalgrowthestimatesshowthattherewereinterestingcohorteffects,withCohort3averyhighachievingcohortinbothELAandmathematics.ThesecohorteffectsareworthyoffurtherstudyandimplythatcautionshouldbeusedwheninterpretinggrowthestimatesfortheORExtforspecificapplications.Whenaveragingacrosscohorts,studentsinELAachievedaRITscoreof206.21pointsinGrade3andgrewtoaRITscoreof219.38byGrade8.Theaveragegrowthwas2.10RITscorepointsperyear.Whenaveragingacrosscohorts,studentsinMathachievedaRITscoreof193.72pointsinGrade3andgrewtoaRITscoreof205.51bytheGrade8.Theaveragegrowthwas2.19RITscorepointsperyear.CurvilinearityisnotedintheELAdata,however,withmoregrowthoccurringattheearliergradesthanatthelatergrades.Mathematicsgrowthappearstobemorelinear.ORoraChangeScoresfrom2016to2017TheORoratotalrawscoresfrom2016and2017werecomparedtodeterminehowmuchchangewasexhibitedfromthefirstadministrationoftheORorain2016tothesecondadministrationin2017.Atotalof849studentsparticipatedintheORorain2016andatotalof772participatedin2017.Only473ofthosestudentsparticipatedintheORoraforbothyearsoftheadministration.Then-sizefortheplotsbelowincludesthose473students.TherangeofpossiblescoresontheORoraisfrom20to80.Themeanscorein2016was46.12,whilein2017themeanwas48.08.Theaveragechangefrom2016to2017ontheORorawas1.827points,buttherewasgreatvariationinchangescores(min=-60,max=+40).
2016 ORora Results
ORora Total Score
Frequency
20 30 40 50 60 70 80
010
2030
40
Mean Score (46.12)
2016-2017 OR Extended Assessment Critical Element 4 – Page
112
2017 ORora Results
ORora Total Score
Frequency
20 30 40 50 60 70 80
010
2030
40
Mean Score (48.08)
ORora Change Scores 2016 to 2017
Delta
Frequency
-60 -40 -20 0 20 40
020
4060
80100
Average Change Score (1.827)
2016-2017 OR Extended Assessment Critical Element 4 – Page
113
4.3FullPerformanceContinuumTheORExtisdesignedtosampletheCommonCoreStateStandardsinEnglishlanguagearts(Reading,Writing,andLanguage)andMathematics,aswellastheOregonScienceStandardsandNextGenerationScienceStandardsinscienceinapurposive,validatedmanner.TheORExttestblueprintsconveythebalanceofrepresentationexhibitedbytheassessment(seeAppendix2.1B).ThesetestblueprintsaresupportedbytheORExtExtendedAssessmentFrameworks(http://www.brtprojects.org/publications/training-modules),whichdefinetheassessablecontentontheORExtthathasbeenreducedindepth,breadth,andcomplexity(RDBC)usingourdefinedprocess(seeAppendix2.3A.3).Thedecisionsregardingwhichstandardstotargetforessentialization,aswellasthestrengthoflinkagebetweentheEssentializedStandardsandtheCCSS/ORSci/NGSShasbeenvalidatedbyOregonteachers,aswell(seeAppendix3.1A).Thoughasimplifiedandstandardizedapproachwastakentodesignitems,andefficiencyandaccesstotheassessmentincreasedforthemajorityofstudents(asevidencedbythedecreasedpercentagesofzeroscoresacrossallcontentareas),asmallsubgroupofstudentsremainswhocannotaccessanacademicassessment.ThisistrueeventhoughitemshavebeensignificantlyRDBCatthreelevelsofcomplexity(low-medium-highdifficulty).Asaresponse,ODEcommissionedBRTtodesignandimplementanobservationalratingscaleforthisgroupofverylow-performingstudents,calledtheOregonObservationalRatingAssessment(ORora)forthespring2016administration.TheORoratargetscommunication(expressiveandreceptive)andbasicskills(attention/jointattentionandmathematics)andprovidesdocumentationofstudentprogressoutsideofourclearlydefinedacademicdomains.Itemsonallassessmentswerescoredona2-pointscale,with1pointawardedforacorrectresponseand0pointsawardedforanincorrectresponse.Plotsareprovidedbelowforeachcontentareaandgradelevel,includingthepersonabilityanditemdifficultydistributions.Ingeneral,thedescriptivestatisticssuggestthatthetesthadanappropriaterangeofitemdifficultiesrepresented,fromeasytodifficult,withitemdifficultiesgenerallyrangingfrom-4.0to+4.0ontheRaschscale.TheassessmentsperformedasexpectedacrossallgradesandcontentareaswiththeexceptionofGrade7mathematics,asnotedabove.TheitempersondistributionsprovidedbelowdemonstratethattheORExtisprovidingaperformancecontinuumforstudentswhoparticipate.
2016-2017 OR Extended Assessment Critical Element 4 – Page
114
EnglishLanguageArtsPerson/ItemDistributions
2016-2017 OR Extended Assessment Critical Element 4 – Page
115
2016-2017 OR Extended Assessment Critical Element 4 – Page
116
2016-2017 OR Extended Assessment Critical Element 4 – Page
117
MathematicsPerson/ItemDistributions
2016-2017 OR Extended Assessment Critical Element 4 – Page
118
2016-2017 OR Extended Assessment Critical Element 4 – Page
119
2016-2017 OR Extended Assessment Critical Element 4 – Page
120
2016-2017 OR Extended Assessment Critical Element 4 – Page
121
SciencePerson/ItemDistributions
2016-2017 OR Extended Assessment Critical Element 4 – Page
122
PersonAbilityandItemDifficultyTablesEnglishlanguagearts PersonAbility ItemDifficulty
Grade Mean SD Mean SD3 0.96 0.56 0.41 0.114 1.30 0.60 0.54 0.125 1.38 0.65 0.57 0.126 1.66 0.64 0.80 0.137 2.06 0.65 0.96 0.138 1.94 0.65 1.20 0.1311 2.20 0.84 0.41 0.11
2016-2017 OR Extended Assessment Critical Element 4 – Page
123
Mathematics PersonAbility ItemDifficulty
Grade Mean SD Mean SD3 -0.38 0.51 -0.41 0.114 -0.15 0.50 -0.23 0.115 -0.16 0.53 0.25 0.116 0.34 0.49 0.58 0.117 0.51 0.53 0.05 0.128 0.60 0.49 1.02 0.1111 0.31 0.49 0.23 0.10Science PersonAbility ItemDifficulty
Grade Mean SD Mean SD5 1.37 0.66 0.60 0.128 1.80 0.74 0.53 0.1411 1.94 0.79 0.33 0.134.4ScoringAllscoringexpectationsfortheORExtareestablishedwithintheAdministrationManual(seeAppendix2.3,p.14).ThescoringproceduresforthenewORExthavebeensimplified,withstudentsreceivinga0foranincorrectresponseora1foracorrectresponse.InputfromthefieldgatheredfromConsequentialValiditystudiesdemonstratesthattheassessmentscoringproceduresaremuchmoreclearandeasiertoimplementthanpriorscoringapproaches(seeAppendix2.3B.10).BRTwasalsocommissionedtodevelopascaledscoreinterpretationguide,whichdescribesspecificstrategiesforinterpretingstudenttestscoresandsub-testscoresinReadingandWriting,andAchievementLevelDescriptors(ALDs)publishedwithintheIndividualStudentReports(seeAppendix6.4C)forannualperformance,growth,andaspartofEssentialSkillsrequirementsforverylowperformingstudents(seeAppendix2.1A).4.5MultipleAssessmentFormsTheORExtwasadministeredinonlyonegradelevelformforthe2016-17schoolyear,with36operationalitemsarrangedinorderofempiricaldifficultyand12embeddedfieldtestitems. 4.6MultipleVersionsofAnAssessmentTheORExtisprovidedinthestandardformat,butisalsoavailableinLargePrintandBrailledformats.Testcontentisidenticalacrossallthreeversions,withanoccasionalitembeingeliminatedontheBrailleversionduetoinaccessibility.Theseitemsdonotcountfororagainstthestudentinreporting.Substantivetestcomparabilityanalysesarenotfeasible,giventhesmalln-sizesofthesamplesinvolvedinthealternativeversions.
2016-2017 OR Extended Assessment Critical Element 4 – Page
124
4.7TechnicalAnalysesandOngoingMaintenanceTheORExttechnicalanalysesthatdocumentreliabilityandvalidityareincludedinthistechnicalreport(seeSections3and4,respectively).ODEandBRTstaffreviewstheseanalysesannually.Necessaryadjustmentstotheassessmentaredeterminedpriortoimplementationofthesubsequentyear'sworkplan,whichelaboratestheareasofimprovementaswellasaspectsofthetestingprogramthatwillbemaintained.Thisdecision-makingissupportedbyinputfromthefieldgatheredfromtheConsequentialValiditystudy(seeAppendix2.3B.10).Onenoteworthyexampleoftheimpactofoursystemofongoingimprovementthisyearisthedevelopmentofadditionalcurricularandinstructionalresources,whichaddressesanareaofconcernexpressedbystakeholders.ThetrainingmoduleswedevelopedtoconnecttheassessmentresultsgarneredfromtheORExtandORorawithcurricularresourcesandinstructionalstrategiesthatarealigned.
2016-2017 OR Extended Assessment Critical Element 5 – Page
125
CriticalElement5–InclusionofAllStudents
5.1ProceduresforIncludingSWDsTheOregonassessmentsystemprovidesexplicitguidanceregardingtheparticipationofallpublicschoolstudentsinitsstatewideassessmentprogram(seeSection1.4).
5.1AClearExplanationsoftheDifferencesBetweenAssessments
TheassessmentoptionsforallpublicschoolstudentsinOregonareelaboratedintheOregonTestAdministrationManual(seeAppendix1.4.2,p.7).TheseoptionsincludetheSmarterBalancedAssessmentinEnglishlanguageartsandmathematicsinGrades3-8&11,theOregonAssessmentofKnowledgeandSkillsinscienceinGrades5,8,&11,andinthesamecontentareasandgradelevelsforSWSCDwhotaketheORExt(seeAppendix1.4.2,p.92-93).SocialstudiesassessmentisadistrictoptionwithintheOAKSportal,aswell.Inaddition,expectationsfortheEnglishLanguageProficiencyAssessment(ELPA)andtheKindergartenAssessmentareprovided.
5.1BEligibilityDecisionsMadebyIEPTeamsAstudent'sIEPteamdetermineshowastudentwithdisabilitieswillparticipateintheOregonStatewideAssessmentprogram.TheIEPteammustaddresstheeligibilitycriteriaforparticipationintheORExtbeforedeterminingthattheassessmentistheappropriateoption(seeAppendix5.1B).
5.1CGuidelinesforAssessmentSelectionAsnotedearlier,IEPteamsmakedecisionsregardinghowstudentswithdisabilitiesparticipateintheOregonstatewideassessmentprogram.Atpresent,studentsparticipateinoneofthreeoptions:(a)studenttakesthegeneralassessmentwithorwithoutuniversaltools.(b)studenttakesthegeneralassessmentwithdesignatedsupportsand/oraccommodations,or(c)studenttakestheORExt.Guidelinesformakinguniversalsupport,designatedsupport,andaccommodationsdecisionsforthegeneralassessmentsareprovidedinAppendix2.3A.1.GuidelinesformakingthesedeterminationsforSWSCDwhoparticipateinAA-AAASareprovidedinAppendix5.1B.
5.1DInformationonAccessibilityOptionsInformationregardingaccessibilityoptionsforthegeneralassessmentcanbefoundwiththegeneralassessmentPeerReviewevidence.FortheORExt,accessibilityistreatedholistically,withuniversaldesignforassessmentconceptsembeddedintheitemdesignandawidevarietyofaccommodationsalsoavailableifneeded.Itemsarecraftedtobevisuallysimpleandclean.Graphicsupports,whicharealwaysblack/whitelinedrawings,areembeddedinallitemsatthelowlevelofcomplexitybutarephasedoutasitemsbecomemorecomplex.Itemsaredesignedtoincorporatesimplifiedlanguageunlessspecificacademicvocabularyandconceptsiswhatisbeingtested(seeAppendix2.3A.3).TheitemsontheORExtareallselectedresponse,withthreeresponseoptionsallowingformultiplemodesofaccess(e.g.,sayingtheanswer,pointingtotheanswer,eyegaze,switch,etc.).Alltextpresentedtostudentsisatleast18-ptfont(larger,ofcourse,inthelargeprintversion).
2016-2017 OR Extended Assessment Critical Element 5 – Page
126
SampleitemsarepresentedinAppendix2.2.3.Allaccessibilitysupports,designatedsupports,andaccommodationsfortheORExtarepublishedinAppendix2.3A.1,p.36-43.ForstudentswhohaveverylimitedtonocommunicationandareunabletoaccesseventhemostaccessibleitemsontheORExt,anOregonObservationalRatingAssessment(ORora)wasimplementedin2015-16.TheORoraiscompletedbyteachersanddocumentsthestudent'slevelofcommunicationcomplexity(expressiveandreceptive),aswellaslevelofindependenceinthedomainsofattention/jointattentionandmathematics.Theadministrationinstructionsand2015-16resultsfortheORoraareincludedinAppendix5.1D.
5.1EGuidanceRegardingAppropriateAccommodationsGuidanceregardingappropriateaccommodationsispublishedinAppendix2.3A.1.DistrictandSchoolTestCoordinatorsprovideannualtrainingontestsecurityandadministration.TheORExtapproachesaccessaspartoftestdesign,asnotedaboveinSection5.1D.ThecomplexityofSWSCDcommunicationsystemsdemandssuchanapproach.Inaddition,comprehensiveaccommodationsareallowedinordertodecreasethechancesthatadisabilitymayinterferewithourabilitytomeasurethestudent'sknowledgeandskills.
5.1FAllSWDsEligiblefortheORExtODE'seligibilityguidelinesmakeitclearthatallSWDsareeligiblefortheORExt,regardlessofdisabilitycategory,andthatspecificdisabilitycategorymembershipshouldnotbeadeterminingfactorforconsideringparticipation(seeAppendix5.1B).
5.1GParentsInformedofAA-AAASConsequencesTheParentFAQsectionoftheGeneralAdministrationManualmakesitclearthatparentsmustbeinformedofthepotentialconsequencesofhavingtheirchildassessedagainstalternateachievementstandards,includingdiplomaoptions.Parentsarealsoinformedthatalternateachievementstandardsaredesignedtoreflectasignificantreductionindepth,breadth,andcomplexityandarethereforenotcomparabletogeneralacademicachievementstandards(seeAppendix2.3,p.28-32).
5.1HStateEnsuresORExtPromotesAccesstotheGeneralEducationCurriculumTheORExtisstronglylinkedtotheCCSS/ORSci/NGSS,asevidencedbyourlinkagestudyresults(seeAppendix3.1A).Theclaimisbasedonthefollowingwarrants:(a)ORExtitemsarealignedtotheEssentializedStandards;(b)theEssentializedStandardsarestronglylinkedtothegradelevelcontentstandards;therefore(c)theORExtitemsarestronglylinkedtogradelevelcontentexpectations.ItisthusexpectedthattheORExtpromotesaccesstothegeneraleducationcurriculumbyassessinggeneraleducationcontentthathasbeenreducedindepth,breadth,andcomplexityyetmaintainsthehighestpossiblestandardforSWSCD.Inaddition,ODEcommissionedBRTtoworkwithOregonteachersofSWSCDinthe2015-16schoolyeartodevelopavarietyofcurricularandinstructionalresourcesthatarealignedtotheEssentializedStandards.Theseresourcesinclude:(a)curriculartemplates,
2016-2017 OR Extended Assessment Critical Element 5 – Page
127
(b)videotutorials,and(c)supportingdocumentsthatprovidespecificguidanceregardinghowtodeveloplessonplans,PresentLevelsofAcademicandFunctionalPerformance(PLAAFP)statements,andIndividualizedEducationProgram(IEP)goalsandobjectivesthatarealignedwiththeEssentializedStandards.Itisalsoexpectedthattheessentializationprocesswillgeneralizetomanystudentswhoareperformingoffgradelevel,notmerelytoSWSCD.AllresourcesarepublishedonaBRT-sponsoredwebsiteathttp://lms.brtprojects.org.5.2A–5.2CProceduresforIncludingELsInadditiontotheprogrammaticguidanceprovidedinAppendix1.4A.1relatedtoELprogrameligibilityandservices,ODEalsoprovidesguidancerelevanttotheinclusionofELsinthestatewideassessmentprograminAppendix1.4.2.ThoughtheORExtiscurrentlypublishedinEnglish,anappropriatelyqualifiedinterpretercanprovidetheassessmenttoanySWSCDfromdiverselanguagebackgrounds,includingAmericanSignLanguage.ODEhasdevelopedatrainingmoduletoincreasethestandardizationofASLadministrationforitsstatewideassessments,availableathttp://lms.brtprojects.org.AdditionalinformationregardingtheinclusionofELsinOregon'sgeneralassessmentsisprovidedinthegeneralassessmentPeerReviewevidence.5.3AccommodationsAllstatewideaccommodationguidanceispublishedintheAccessibilityManual(seeAppendix2.3A.1),outliningtheuniversaltoolsanddesignatedsupportsavailabletoallstudents,andaccommodations,availableonlytostudentswithdisabilitiesorstudentsservedbySection504Plans.Inaddition,themanualdefinesthesupportsasembedded,wheretheyareprovidedbytheonlinetestengine(e.g.,calculator,text-to-speech),ornon-embedded,wheretheymustbeprovidedbyaqualifiedassessor(e.g.,readaloud,scribe).Themanualalsomakesitclearthatthesesupportsarecontent-areaspecific,asauniversaltoolinonecontentareamaybeanaccommodationinanother.
5.3AAppropriateAccommodationsareAvailableforSWD/Section504AppropriateaccommodationsfortheORExtarepublishedinAppendix2.3A.1,p.36-43.Additionalaccommodationsforallstatewideassessmentsarealsopublishedinthismanual.TheOregonAccommodationsPanelreviewstheappropriatenessofthesupportslistedannually.Practitionersmayalsorequesttheadditionofanaccommodationthroughaformalprocess(seeAppendixE:ApprovalProcessforNewAccessibilitySupportswithinthemanual,Appendix2.3A.1,p.100-102).
5.3BAppropriateAccommodationsareAvailableforELsAsnotedinSections5.2A-C,theORExtisaccessibleinanycommunicationmodalitythroughtheuseofaninterpreter.AppropriateaccommodationsfortheORExtarepublishedinAppendix2.3A.1,p.36-43.Additionalaccommodationsforallstatewideassessmentsarealsopublishedinthismanual.TheOregonAccommodationsPanelreviewstheappropriatenessofthesupportslistedannually.Practitionersmayalsorequestthe
2016-2017 OR Extended Assessment Critical Element 5 – Page
128
additionofanaccommodationthroughaformalprocess(seeAppendixE:ApprovalProcessforNewAccessibilitySupportswithinthemanual,Appendix2.3A.1,p.100-102).
5.3CAccommodationsareAppropriateandEffectiveInadditiontotheevidencegatheredduringthelinkagestudy(seeAppendix3.1A),whichsuggeststhattheORExtitemswereaccessibleandfreeofbiasevenbeforefinalediting,theappropriatenessofthesupportslistedinAppendix2.3A.1isreviewedannuallybytheOregonAccommodationsPanel.Practitionersmayalsorequesttheadditionofanaccommodationthroughaformalprocess(seeAppendixE:ApprovalProcessforNewAccessibilitySupportswithinthemanual,Appendix2.3A.1,p.100-102).ODEiscollectingaccommodationscodesfortheORExtfromQualifiedAssessorswhoopttoenterthisinformationinordertomakeperformancecomparisonsfeasible.Itishopedthatthisprocesswillberequiredbyspring2018.Theconsequentialvaliditystudyfor2018willincludequestionsregardingtheappropriatenessoftheavailableaccommodations,aswell.
5.3DAccommodationsareAppropriateandEffectiveODEhasaformalprocessstakeholderscanusetorequestaccommodationsthatarenotalreadypublishedintheAccessibilityManual(seeAppendixE:ApprovalProcessforNewAccessibilitySupportswithinthemanual,Appendix2.3A.1,p.100-102).5.4A–5.4EMonitoringTestAdministrationforSpecialPopulationsODEmonitoringoftestadministrationinitsdistrictsandschoolsiselaboratedwithinthegeneralassessmentPeerReviewevidenceandisthereforenotaddressedhere.
2016-2017 OR Extended Assessment Critical Element 6 – Page
129
CriticalElement6–AcademicAchievementStandardsandreporting6.1StateAdoptionofAlternateAcademicAchievementStandardsforSWSCDTheOregonExtendedassessment(ORExt),Oregon'sAlternateAssessmentbasedonAlternateAcademicAchievementStandards(AA-AAAS),ispartoftheOregonStatewideAssessmentSystem.TheORExtisadministeredtoOregonstudentswiththemostsignificantcognitivedisabilities(SWSCD)inEnglishlanguageartsandmathematicsinGrades3-8and11.TheORExtisadministeredinscienceinGrades5,8,&11.TheORExtlinkstotheCCSSinEnglishlanguageartsandmathematics.ThenewORExtisduallylinkedtoOregon'sformersciencestandards,aswellastotheNGSS.ResultsfromtheEnglishlanguageartsandmathadministrationsareincludedincalculationsofparticipationandperformanceforAnnualMeasureableObjectives(AMO)–aprovisionoftheNoChildLeftBehindAct(NCLB).ScienceparticipationisalsoincludedaspartoftheTitle1AssessmentSystemrequirements,andisadministeredingrades5,8,&11.TherevisedORExtisbuiltuponaverticalscaleinordertosupportreliabledeterminationsofannualacademicgrowthinELAandmathematicsinGrades3-8.ThecompleteverticalscalingplanandoperationalitemselectiondecisionrulesarelocatedinAppendix2.2.1.
6.1AStateFormallyAdoptedAlternateAcademicAchievementStandards
TheStateBoardofEducationformallyadoptedtheAAASandachievementleveldescriptors(ALDs)onJune25,2015(seeAppendix6.1A.1).TheELA,Math,andScienceAAAS,includingboththeALDsandtherequisitecutscoresareincludedinAppendix6.1.A.2.
6.1BStateAppliesAAAStoAllPublicSchoolSWSCDinTestedGrades
ThestateappliestheAAAStoallpublicschool-servedSWSCDwhoparticipateintheORExtinGrades3-8&11inEnglishlanguageartsandmathematics,andinGrades5,8,&11inscience.
6.1CState'sAAASIncludeAtLeastThreeLevels,ALDs,andCutScoresThealternateacademicachievementstandardsinOregonarecomposedoffourlevels(thoughonlythreearerequired).Indescendingorder,theyare(a)Level1,(b)Level2,(c)Level3,and(d)Level4.Level1andLevel2performancesrepresentproficientachievement,whilethebottomtwolevelsrepresentachievementthatisnotyetproficient.TheproceduresfollowedtodevelopOregon'salternateacademicachievementstandardswereconsistentwithTitle1assessmentsystemrequirements,includingtheestablishmentofcutscores,whererelevant.Inordertodefinefourlevelsofproficiency,Oregonsetthreecutscoresacrossallsubjectareas:(a)toseparateLevel1fromLevel2,(b)toseparateLevel2fromLevel3,and,(c)toseparateLevel3fromLevel4.ThealternateacademicachievementstandardsinEnglishlanguagearts,mathematics,andsciencefortheORExt,includingtheachievementleveldescriptors(ALDs)andcutscores,wereestablishedduringstandardsettingmeetingsheldonJune15(science),16(mathematics),and17(Englishlanguagearts).
2016-2017 OR Extended Assessment Critical Element 6 – Page
130
6.2AchievementStandardSettingStandardSettingmeetingswereheldattheUniversityofOregoninEugene,ORonJune15,2015(Science),June16,2015(Mathematics),andJune17,2015(Englishlanguagearts).Atotalof53standardsetterswereinvolvedintheprocess:11inScience,and21inbothEnglishlanguageartsandMathematics.Panelistswereassembledingradelevelteamsofthree,wheretwomemberswerespecialeducatorsandonememberwasacontentspecialist.Thepanelistswerehighlyeducated.Over90%ofthepanelpossessedaMaster’sdegreeorhigher.Fifty-seven(57%)percentofthepanelistshadover11yearsofteachingexperience.Seventy-sixpercent(76%)ofthepanelistshadsomeexperienceworkingwithstudentswithsignificantcognitivedisabilitieswith64%licensedasSpecialEducators.Themajorityofpanelmemberswerefemale(87%),fromtheNorthwestofthestate(87%),andWhite(83%).Nopanelmemberself-identifiedwithOregon’smajorminoritypopulation(Hispanic).Inadditiontothelivetrainingduringstandardsettingmeetings,panelistswereaskedtocompleteseveraltrainingrequirementspriortothestandardsettingmeetings,whichorientedthemtothestudentpopulationofstudentswithsignificantcognitivedisabilities(SWSCDs),theOregonExtendedAssessmenttestdesignandhistory,aswellasthebookmarkingstandardsettingmethod.Panelistswerequiteconfidentintheirpreparationandfinaljudgments,asevidencedbyresponsestothequestions:(a)"Thetraininghelpedmeunderstandthebookmarkmethodandhowtoperformmyroleasastandardsetter."(b)"Iamconfidentaboutthedefensibilityandappropriatenessofthefinalrecommendedcutscores."and,(c)"Overall,IamconfidentthatthestandardsettingproceduresallowedmetousemyexperienceandexpertisetorecommendcutscoresfortheORExt."Theheartymajorityofstandardsettersstronglyagreedwiththesestatements,whileallparticipantsagreed.Thenine-stepprocessimplementedforthesestandardsettingmeetingswasbasedonHambleton&Pitoniak(2006)asreportedbyR.L.Brennan(EducationalMeasurement,4thEdition,pp.433-470).StandardsettingevaluationquestionsposedtoparticipantswereadaptedfromCizek'sSettingPerformanceStandards(2012).StandardsetterssetcutscoresandrecommendedAchievementLevelDescriptors(ALDs)fortheOregonStateBoardofEducationtoconsider.Thecutscoreswerearticulatedtoreflectverticaldevelopment,oratleastmaintenance,ofexpectationsacrossgradesinamannerthatrespectedstandardsetterjudgmentstothegreatestpossibledegree.SixchangesweremadeinELAandMathematics.Scienceisnotbuiltuponaverticalscale,sonocutscoreadjustmentswerenecessaryinScience.Thecutscoresarelistedbelow.
2016-2017 OR Extended Assessment Critical Element 6 – Page
131
Englishlanguagearts(ELA)
Grade Level1 Level2 Level3 Level43 191orbelow 192-212 213-227 228orabove4 199orbelow 200-212 213-227 228orabove5 201orbelow 202-219 220-231 232orabove6 204orbelow 205-219 220-232 233orabove7 207orbelow 208-221 222-235 236orabove8 212orbelow 213-223 224-235 236orabove11 898orbelow 899-919 920-926 927orabove
Mathematics
Grade Level1 Level2 Level3 Level43 191orbelow 192-200 201-217 218orabove4 192orbelow 193-205 206-218 219orabove5 192orbelow 193-205 206-219 220orabove6 203orbelow 204-207 208-221 222orabove7 206orbelow 207-208 209-222 223orabove8 207orbelow 208-211 212-225 226orabove11 900orbelow 901-906 907-921 922orabove
Science
Grade Level1 Level2 Level3 Level45 505orbelow 506-516 517-529 530orabove8 809orbelow 810-819 820-830 831orabove11 900orbelow 901-913 914-928 929orabove
Note:TheELAandMathverticalscalesfortheORExtarecenteredon200ingrades3-8andcanbeusedtodocumentyear-to-yeargrowth.Noneoftheotherscalesshouldbeusedforlongitudinalcomparisons.AllGrade11scalesareindependentandcenteredon900.Thegrade5Sciencescaleisindependentandcenteredon500,whiletheGrade8Sciencescaleisindependentandcenteredon800.Anindependentauditorevaluatedthebookmarkingstandardsettingprocess.Theauditor'scomprehensivereportcanbefoundinAppendix6.2.2.
6.3ChallengingandAlignedAcademicAchievementStandardsOregoneducatorsinitiallyevaluatednewOregonEssentializedAssessmentFrameworksintworespects.First,educatorswereaskedtodeterminetheappropriatenessofthestandardsselectedforinclusionandexclusionintheEssentializedStandards(yes/no).Second,theleveloflinkagebetweentheEssentializedStandardsandgradelevelcontentstandardwasevaluated(0=nolink,1=sufficientlink,2=stronglink).Summaryresultsareprovidedinthetablesbelow.Acomprehensiveessentializedstandardtogradelevel
2016-2017 OR Extended Assessment Critical Element 6 – Page
132
standardlinkagestudy,aswellasessentializedstandardtoitemalignmentstudy,isprovidedinAppendix3.1A.Englishlanguagearts
Grade#
EssentializedStandards
#Raters Ave.LinkageRating(0-2)*
Ave.AgreementwithEssentialization(0-6)*
3 27(38) 6 1.74(10) 5.68(21)4 30(40) 6 1.78(15) 5.77(25)5 28(39) 6 1.73(12) 5.79(23)6 25(37) 6 1.80(12) 5.76(19)7 24(36) 6 1.77(10) 5.79(19)8 25(36) 6 1.79(12) 5.80(21)11 24(36) 6 1.82(12) 5.79(19)Note.*Countofperfectratings/agreementacrossallraters(inparenthetical)relativetonumberofessentializedstandards.Mathematics
Grade#
EssentializedStandards
#Raters Ave.LinkageRating(0-2)*
Ave.AgreementwithEssentialization(0-3)*
3 22(33) 3 2.00(22) 2.77(17)4 26(34) 3 1.99(25) 2.81(21)5 23(34) 3 1.99(22) 2.78(18)6 27(41) 3 1.98(21) 2.68(15)7 20(36) 3 1.95(17) 2.90(18)8 19(33) 3 1.96(17) 2.37(7)11 23(179) 3 2.00(23) 2.52(12)Note.*Countofperfectratings/agreementacrossallraters(inparenthetical)relativetonumberofessentializedstandards.Science
Grade#
EssentializedStandards
#Raters Ave.LinkageRating(0-2)*
Ave.AgreementwithEssentialization(0-4)*
5 15(16) 4 1.92(10) 3.93(14)8 24(59) 4 1.97(21) 4.00(24)11 24(71) 4 1.98(22) 3.83(20)Note.*Countofperfectratings/agreementacrossallraters(inparenthetical)relativetonumberofessentializedstandards.
2016-2017 OR Extended Assessment Critical Element 6 – Page
133
6.4ReportingOregon'sreportingsystemfacilitatesappropriate,credible,anddefensibleinterpretationanduseofitsassessmentdata.WithregardtotheORExt,thepurposeistoprovidethestatetechnicallyadequatestudentperformancedatatoascertainproficiencyongradelevelstatecontentstandardsforstudentswithsignificantcognitivedisabilities(seeSections3and4).Inaddition,thestatemakesitclearthatresultsfromtheOregonExtendedarenotcomparabletoresultsfromtheSBA/OAKS(seeAppendix2.3,p.29-31).Nevertheless,thetestmeetsrigorousreliabilityexpectations(seeSection4.1).ValidityisconsideredhereasanoverarchingsummationoftheOregonExtendedassessmentsystem,aswellasthemechanismsthatOregonusestocontinuouslyimprovetheORExtassessment(seeAppendix2.3B.10).
6.4APublicReportingOregonreportsparticipationandassessmentresultsforallstudentsandforeachoftherequiredsubgroupsinitsreportsattheschool,district,andstatelevels.Thestatedoesnotreportsubgroupresultswhentheseresultswouldrevealpersonallyidentifiableinformationaboutanindividualstudent.ThecalculationrulefollowedisthatthenumberofstudentsinthesubgroupmustmeettheminimumcellsizerequirementforeachAMOdecision:participation,achievementinEnglishlanguageartsandmath,attendance,andgraduation,whereappropriate(seeAppendix2.6C)
6.4BStateReportsInterpretableResults
Oregondevelopsanddisseminatesindividualstudentdatauponfinaldeterminationofaccuracy.Thestateprovidesdistrictswithindividualstudentreports(ISRs)thatmeetmostrelevantrequirements.ThestateincorporatedtheStandardErrorofMeasure(SEM)foreachstudentscoreintothereporttemplates.TheSEMassociatedwitheachcutscoreisprovidedinSection4.1B.Also,seethemock-upISRinAppendix6.4C.
6.4C1–C5StateProvidesIndividualStudentReports
Oregon'sstudentreportsprovidevalidandreliableinformationregardingachievementontheassessmentsrelativetotheAAS.ThereliabilityofthedataisaddressedinSection4.1.ValidityisconsideredhereasanoverarchingsummationoftheOregonExtendedassessmentsystem,aswellasthemechanismsthatOregonusestocontinuouslyimprovetheOregonExtendedassessment.TheISRsclearlydemonstratethestudents'scalescorerelativetheAAASthatisrelevantforthatcontentareaandgradelevel(seeSection4.4andAppendix6.4C).TheOregonISRsprovideinformationforparents,teachers,andadministratorstohelpthemunderstandandaddressastudent'sacademicneeds.Thesereportsaredisplayedinasimpleformatthatiseasyforstakeholderstounderstand.Districtrepresentativescantranslateresultsforparentsasnecessary.ScaledscoreinterpretationguidanceispublishedinAppendix2.1A.
2016-2017 OR Extended Assessment Conclusions and Next Steps– Page
134
ConclusionsandNextSteps
Insum,therigoroftheproceduraldevelopmentandstatisticaloutcomesoftheORExtweresubstantiveandsupporttheassessmentsintendedpurpose.Proceduralevidenceincludesessentializedstandardsdevelopment,itemdevelopment,itemcontentandbiasreviews,anindependentalignmentstudyanditemselectionbaseduponitemcharacteristics.Outcome-relatedevidenceincludedmeasurereliabilityanlayses,pointmeasurebiserials,outfitmeansquares,itemdifficultyandpersonabilitydistributions,andconvergentanddivergentvalidityevidence.Thesesourcesofevidencewereallquitegoodandprovideimportantvalidityevidence.ThetestdevelopmentprocessadheredtoproceduralguidelinesdefinedbytheAERA/APA/NCMEStandardsforEducationalandPsychologicalTesting(2014),aswellasincorporatingproceduresthatareknowninthefieldtobebestpractice.Forexample,anindependentauditorevaluatedalignment.Inaddition,theORExtreflectswhathighlyqualifiedOregoneducatorsbelieverepresentsthehighestprofessionalstandardsforthepopulationofstudentswithsignificantcognitivedisabilities,asevidencedinourconsequentialvaliditystudybyteachersupportoftheacademiccontentontheORExtaswellasthebehaviorssampledduringtestadministration.Dr.DiannaCarrizalesconductedanindependentalignmentstudyconsistingoffiveevaluationcomponents:a)standardselectionforessentialization,b)strengthoflinkagebetweenessentializedstandardsandgradelevelcontentstandards,c)alignmentbetweenitemsandessentializedstandards,d)alignmentbetweentheessentializedstandardsandtheachievementleveldescriptors,ande)alignmentbetweentheachievementleveldescriptorsandtheORExttestitems.Dr.Carrizalesreportedthat,“Inthethreeevaluationsthatinvolveddeterminingtherelationshipbetweenstandardsanditems,reviewersidentifiedsufficienttostrongrelationshipsamongassessmentcomponentsinallgradesandallsubjectareas.InthetwoevaluationsinvolvingAchievementLevelDescriptors,reviewersidentifiedthirtyinstancesofsufficienttostrongrelationshipsoutofthirty-fourpossiblerelationshipopportunitiesresultinginanoverallaffirmedrelationshipwithareasforrefinementsidentified.”Overall,documentationcollectedinthereportsuggeststhattheORExtassessmentsystemisaligned.ThetestreliabilitiesfortheORExtwerequitehigh,suggestingthattheassessmentitemsfunctionedconsistentlywiththetestasawhole.Thecorrelationsbetweenstudents’contentscoresacrosssubjectswerenotoverlystrong,implyingthateachtestmeasuresadistinctconstruct.TheclassificationconsistencyanalysesdemonstratethattheORExtisappropriatelycategorizingstudentsintotheproficientcategory,andcapableofdoingsoinaconsistentmanner.Theverticalscaledevelopedin2014-15appearstobemodelingincrementalgrowthacrossGrades3-8inELAandmathematics,asintended.TheGrade7mathematicstestcontinuedtodemonstrateinsufficientitemdifficultiesacrosstherangeoflow,medium,andhighitemcomplexity,however,andmustagainbeamendedinthe2017-18schoolyear.TheELAandscienceassessmentscouldcontinuetobenefitfromtheadditionofmoredifficultitems,asevidencedbycomparisonsoftheaverageperson
2016-2017 OR Extended Assessment Conclusions and Next Steps– Page
135
abilitiesanditemdifficulties.Mathematicsassessmentsappeartobefunctioningquitewellintermsofpersonabilitiesanditemdifficulties,thoughsomeadditionallowlevelitemsmighthelpincreaseaccessforthegroupofstudentsfunctioningatthatlevel.TheOregonObservationalRatingAssessment(ORora)resultsdemonstratethatapproximately17-25%oftheSWSCDwhoparticipatedintheORExtalsotooktheORora,dependingupongradelevel.Atotalof755studentswereadministeredtheORorainthe2016-17testadministration.Theparticipantswereprimarilystudentswithmultiple,severedisabilitieswithverylimitedcommunicationsystems.AnalysesofmissingdatapatternsfortheORExtdemonstratedthatQAsweregenerallyabletoadheretothediscontinuationrules.Quantitativeresultsindicatethatatotalof755studentsacrossalltestedgradeswereadministeredtheORora.ResponsepatternsontheORExtwerecomparedtoORoraresultstodeterminewhatpercentagesofQAswereadministeringtheORoraduetotheminimumparticipationruleandwhatpercentagewereadministeringtheORoraoftheirownvolition.Analysesshowedthat234studentswereeligibletotaketheORorainEnglishlanguagearts,241studentswereeligibletotaketheORorainmathematics,and86wereeligibletotaketheORorainscience.Thismeansthatabout30studentspergrade,percontentareareceivedfiveorfewercorrectresponseswithinthefirst15itemsadministeredontheORExt.Ofthe561testrecordsthatmetORoraeligibilityrequirements,91werenotadministeredtheORora.Inaddition,therewere82studentsinELAandMath,respectively,whowereadministeredtheORorawithouthavingparticipatedintheORExt(74ofthosestudentswerethesamestudents,acrosseachcontentarea,witheightstudentsuniquetoeachcontentarea,respectively).The2016-17OregonConsequentialValiditystudyprovidesimportantinformationforfutureadministrationsoftheORExt.TheresultsdemonstratethatthetestcontinuestobeeasytoadministerandscoreandisprovidinganaccessibleandappropriaterepresentationoftheknowledgeandskillsthatshouldberequiredofSWSCDinOregon.Areasofrequestedimprovementincludetheprovisionofatablet-basedadministration,whichisalreadyplannedfor2017-18,andthedevelopmentofadditionallifeskillsitems,whichcannotbeaccomplishedwhilemaintainingrigorousacademicexpectationsthatarelinkedtoOregoncontentstandards.The2016-17OregonExtendedAssessmentPilotTabletAdministrationdemonstratedthatOregonteachershighlyvalueprovisionofatablet-basedadministrationoftheORExtatthestatewidelevel.Benefitsofatablet-basedadministrationincluded:increasedstudentengagement,improvedstandardization,easeofusebyteachers,andresourceprotection(i.e.,time,printing,expense).Theresultsalsosuggestthatmorerobustsystemsareneededtosupportuseraccesstothetestingapplicationviaanautomaticusernameandpasswordprocess.FocusGroupmembersalsorecommendedthatpracticeitemsbedevelopedinatabletformatsoqualifiedassessorsandstudentscanpracticewiththetabletadministrationinpreparationfortheORExttestwindow.Documentingevidenceofvalidityremainsanongoingandcontinuousprocess.Oureffortstocontinuetoimprovetheassessmentsystemareoutlinedbelow,aswellasinSections3and4above.Wealsohavestudiesplannedoverthecourseofthenextthreeyearsthatwill
2016-2017 OR Extended Assessment Conclusions and Next Steps– Page
136
helptosolidifytheevidencethatisaccumulating.AlloftheevidencewehaveathandsuggeststhattheORExtissufficienttoitsstatedpurposeofprovidingreliabledeterminationsofstudentproficiencyatthetestlevelinordertosupportsystemslevelanalysisofdistrictandstateprograms.TheORExtwillhopefullycontinuetoimproveovertimeduetofield-testingandconstantmonitoringandreview,andadditionalvalidityevidencewillbegathered.AsmentionedaboveinSection3.1A,dataarepresentedtosupporttheclaimthatOregon’sAA-AAASprovidesthestatetechnicallyadequatestudentperformancedatatoascertainproficiencyongradelevelstatecontentstandardsforstudentswithsignificantcognitivedisabilities–whichisitsdefinedpurpose.Inthistechnicalreport,wehaveprovidedcontentvalidityevidencerelatedtotheORExttestdevelopmentprocess(i.e.,essentializationprocess,linkagestudy,distributeditemreview,testblueprint,itemwritertraininganddemographics,anditemreviewertraininganddemographics),ORExttestreliabilityevidence,andORExtconsequentialvalidityevidence.FurtheranalysesoverthecomingyearsareplannedtocontinuethedevelopmentoftechnicaldocumentationforoverallconstructvalidityoftheORExt.Thetechnicaldocumentationplanforthe2016through2019schoolyearsisprovidedbelow:DocumentationDescription
AnticipatedTimeframe
Outcome
ScoringAccuracyStudy Spring2018 Provideinter-raterreliabilitydocumentationfortheORExt.
AccommodationsStudy Spring2019 ProvidedocumentationrelatedtotheimpactofaccommodatedtestadministrationfortheORExt.
ORExtEligibilityStudy Fall2019 ProvidedocumentationoftheconsistencyofIEPteamdecisionmakingwiththeestablishedODEguidelinesforORExteligibility.
ORExtFieldTesting Ongoing Testanditemcharacteristicsarereviewedannually,withoperationalitemsthatarenotfunctioningasintendedreplacedbyfieldtestitemsthatarefunctioningproperly.
2016-2017 OR Extended Assessment Appendix Table and Descriptions– Page
137
AppendixTableandDescriptions
AppendixTableTopic FileName
EssentializedAssessmentFrameworksUserGuide App1.1_EAF_UserGuide
IndependentreviewofrigorofCCSSstandards App1.2_FordhamCCSSReview
MemofromtheOregonGovernorregardingparentopt-outoftestingexpectations
App1.4.1_ExecutiveMemo
ODETestAdministrationManual(TAM)App1.4.2_TAM2016_17
ODEEnglishLearnerProgramGuideApp1.4A.1_ODE_ELProgramGuide
Oregonregulationregardinglanguageofassessment App1.4A.2_OAR581_022_0102
Oregon’sannualreporttothestatelegislatureregardingOregon’spublicschools
App1.5_ODEStateReportCard2016
ORExtTestSpecificationsApp2.1_ORExtTestSpecs
ORExtScaledScoreInterpretationGuideApp2.1A_ORExtScaleScoreInterp
ORExtTestBlueprintApp2.1B_ORExt_TestBlueprint_2017
ORExtItemDevelopmentProcessManualApp2.1C_ORExt_ItemDevt_Process
BRTItemWriterTrainingPPTSlidesApp2.2.1_ItemWriter_Training
ORExtVerticalScalingPlanApp2.2.2_ORExtVertScale
ORExtSampleItemsApp2.2.3_ORExtSampleItems
ORExtAdministrationManualApp2.3_ExAssessAdminMan2016_17
ODEAccessibilityManual App2.3A.1_ODEAccessibilityMan2016_17ODEAccommodationsGuide App2.3A.2_OSAAccomGuideBRTProceduresforReducingDepth,Breadth,andComplexityofItems App2.3A.3_ORExtRDBC
ODEQualifiedTrainerPPTSlides App2.3B.1_QATraining2016_17BRTQualifiedTrainerPPTSlides App2.3B.2_ORExtendQTTrng2016_17
2016-2017 OR Extended Assessment Appendix Table and Descriptions– Page
138
Topic FileNameODEStatewideAssessmentTestSchedule App2.3B.4_TestSchd_2016_17QualifiedTrainerSuggestedAgenda App2.3B.5_QT_Training_Agenda2016_17HowtoAccesstheor.k12test.comWebsite App2.3B.6_ExtAssessAccessInstr2016_17QualifiedTrainerResponsibilities App2.3B.7_TrainerResponsibilities2016_17FrequentlyAskedQuestionsDocument App2.3B.8_ORExtFAQ2016_17HelpDeskTechnicalAssistanceLog App2.3B.9_HelpDeskLog2016_17ConsequentialValidityStudyReport
App2.3B.10_ORExtCVStudy2016_17
ORExtPilotTabletStudyReport–Phase2App2.3C_ORExtPilotTabletStudyReport
DataEntryGuideApp2.6B_G3-8_11_DataEntryGuide2016_17
ReportCardRatingPolicyTechnicalManual App2.6C_ReportCardRating_PolicyTechManual
IndependentORExtItemLinkage,Bias,andSensitivityStudyReport App3.1A_LinksAlignmentORExtSpring2017
OverviewoftheDistributedItemReviewwebsiteusedforconductingalignment,bias,andsensitivityreviews
App3.1B_DIR_SystemIntro
HistoryoftheORExtDocumentApp4.1B_HistoryORExt_2017
ReportconveyingtheethnicdiversityofOregon’sstudentsforthe2014-15schoolyear
App4.2_FallMembershipReport2016_17
ODEEligibilityGuidancefortheORExtApp5.1B_ORExtEligibilityGuidance
Resultsfromthe2016-17administrationoftheORoraandadministrationandscoringinstructions.
App5.1D_OroraResults2016_17
AgendaandminutesfromtheStateBoardadoptionoftheAAASfortheORExt App6.1A.1StateBoardAAASAdoption
AllAAASfortheORExt,includingALDsandcutscores App6.1A.2_ORExtAAAS
StandardSettingTrainingPPTSlidesApp6.2.1_ORExt_Standard_Setting
IndependentStandardSettingAuditReportApp6.2.2_ORExtSSAuditRept
ORExtIndividualStudentReportMock-UpApp6.4C_ORExt_ISR
2016-2017 OR Extended Assessment Appendix Table and Descriptions– Page
139
AppendixDescriptions
Appendix1.1Appendix1.1explainsthedevelopmentprocessandintendedusesfortheEssentializedAssessmentFrameworks(EAFs).TheEAFsaretheessentializedstandards(EsSt),whicharelinkedtogradelevelcontentstandards.TheORExtisalignedtotheEAFs,aswell.WhiletheEAFsprimarilyguideitemdevelopment,theyarealsointendedtobeusedinthedevelopmentofappropriatePresentLevelsofFunctionalandAcademicPerformance(PLAAFP)statementsandIndividualizedEducationProgram(IEP)goalsandobjectives.
Appendix1.2
Appendix1.2conveystheevaluationconductedbyresearchersattheFordhamInstitute,whichcomparedthen-currentstatestandardstotheCCSSintermsofrigor.ThefindingsgenerallyshowthattheCCSSareasrigorousormorerigorousthanstatestandards.
Appendix1.4.1Appendix1.4.1istheExecutiveMemofromtheGovernorofOregonregardingparentopt-outexpectations.
Appendix1.4.2Appendix1.4.2isthetestadministrationmanual(TAM)forallassessmentsintheOregonstatewideassessmentsystem,includingtheSBA,OAKS,theORExt,theKindergartenAssessment,andtheELPA.TheTAMelaboratesallrelevanttestsecurityandadministrationprocedures.
Appendix1.4A.1Appendix1.4A.1isODE'sEnglishLearnerProgramGuide,outliningEnglishlearner(EL)systemrequirementsintheareasofstudentidentification,services,reporting,andassessmentforELsinOregon'spublicschools,includingELswhoareSWD.
Appendix1.4A.2
Appendix1.4A.2isOregon’sregulationsthatrequireODEtoprovidetranslatedOAKSassessmentsforpopulationsatorabove9%ingradesK-12withinthreeyearsaftertheschoolyearinwhichthelanguageexceedsthethreshold.
Appendix1.5Appendix1.5isOregon’sannualreporttothestatelegislatureforthe2015-16schoolyear.Thereportincludesstudentdemographicsandinformationonstudentgroups,schoolfundingandstaffinformation,testresults,graduationanddropoutrates,charterschooldataandinformationonalternativeeducationprograms,earlychildhooddata,andattendanceandchronicabsenteeismdata.
2016-2017 OR Extended Assessment Appendix Table and Descriptions– Page
140
Appendix2.1Appendix2.1isthetestspecificationsdocumentthatdescribesourapproachtoassessmentandtestdesignfortheORExt.ThedocumentincludesourapproachtoRDBC,anoverviewoftheessentializationprocessandEAFdocuments,theanticipatedoperationaltestdesignfortheORExt,testdevelopmentconsiderations,sampletestitems,itemspecifications,anduniversaltools/designatedsupports/accommodations.
Appendix2.1AAppendix2.1AprovidesthefieldwithcomprehensiveinformationrelatedtoscaledscoreinterpretationfortheORExt.Theguidanceispublishedinthreemainareas:1)Annualperformance,2)Annualgrowth,and3)Performanceforverylowfunctioningstudents.Guidanceregardinguseandinterpretationofreadingandwritingsubscoresisalsoprovided.
Appendix2.1BAppendix2.1BisthetestblueprintfortheORExt,conveyingthebalanceofrepresentationofdomainsacrossthecontentareasandgradelevelsassessed.Operationalitemsareselectedtoreflecttherepresentationpercentagesincludedinthetestblueprint.
Appendix2.1CAppendix2.1Cdescribestheeight-stepitemdevelopmentprocessusedtodevelopitemsfortheORExt,fromstandardselectiontotestbookletformation.Theitemdevelopmentprocessisspecificandexplicitinordertoincreasetransparency.
Appendix2.2.1Appendix2.2.1isthesetofPPTslidesthatwereusedtotrainitemwritersfortheORExt.Itemwriterswerealsoprovidedanorientationtothetestspecificationsaspartoftraining.
Appendix2.2.2Appendix2.2.2isadocumentthatsummarizesthebalanceddesignverticalscalingplanemployedfortheORExtinthe2014-15administration.Thedocumentincludesthedomainsamplingplanforallassessments,aswellasthedecisionrulesemployedtoremoveitemsfromtheoperationalitempoolpriortoverticalscalingandstandardsettingprocedures.
Appendix2.2.3Appendix2.2.3providesstakeholderswithvisualrepresentationofthestructureoftheORExt.SampleitemsareconveyedinEnglishlanguagearts,mathematics,andscience,withthescoringprotocolandstudentmaterialspresentedtogether.Stakeholderscanseethestructureofeachitem,aswellashowtheitemsarescored.Theycanalsogatheranideaaboutthetypesofformatsthatareusedforanswerchoicesthatareincludedwithinthestudentmaterialsdocuments.
2016-2017 OR Extended Assessment Appendix Table and Descriptions– Page
141
Appendix2.3Appendix2.3isODE'sGeneralAdministrationandScoringManualfor2016-17.ThemanualestablishesODE'sexpectationsregardingthetestwindow,utilizingtheORExttrainingandproficiencywebsite,usingthesignlanguageinterpretertrainingandproficiencywebsite,andinformingparents.Italsoprovidesthefollowinginformationforstakeholders,includingeducatorsandparents:
• OverviewoftheExtendedAssessments• AssessingaStudent• Scoring• DecisionMaking• InformationforTeachers
Themanualprovidesthreeappendicesthatprovideguidanceregardingtheprovisionofsupports,parentquestionsandanswers,andaglossary.
Appendix2.3A.1Appendix2.3A.1isthe2016-17accessibilityoptionsmanualforallassessmentsintheOregonstatewideassessmentsystem,includingtheSBA,OAKS,theORExt,andtheELPA.OptionsincludeUniversalTools,DesignatedSupports,andAccommodations.Themanualprovidesguidanceregardinguseoftheseoptionsininstructionandassessment,aswellasimplementationstrategiesanduseevaluation.EachaccommodationiscodedforuseindataanalysisrelatedtoassessmentscoresfortheSBAandOAKS.
Appendix2.3A.2Appendix2.3A.2isODE'sHowtoSelect,Administer,andEvaluateAccommodationsonOregon'sStatewideAssessmentmanualfor2013-14.Themanualtrainsusersregardinghowtoimplementandevaluateappropriateaccommodations,fromthestudentleveltothesystemslevel.
Appendix2.3A.3Appendix2.3A.3isadocumentthatsummarizestheproceduresusedduringitemdevelopmenttoreduceitemdepth,breadth,andcomplexity,inadditiontothetestspecificationsinformationfoundinAppendix2.1.Thedocumentalsoprovidesmoredetailregardinghowlanguagecomplexityisaddressedandreviewedinanefforttodecreasethelanguageloadofitemsandmakethetestmoreaccessibletoallstudents.Thedocumentalsodiscusseswaysinwhichbiasisaddressedduringtestdevelopment.
2016-2017 OR Extended Assessment Appendix Table and Descriptions– Page
142
Appendices2.3B.1-2.3B.2Appendices2.3B.1and2.3B.2arethePowerPoint(PPT)trainingsthatwereusedbyODEandBRTtrainerstotrainnewqualifiedassessors(QAs)andqualifiedtrainers(QTs)infourregionallyhostedtrainingsinNovember2016.QTsalsousedthepackagetotrainNewQualifiedAssessorsforthe2016-17schoolyear.ThetrainingprovidesparticipantswiththeinformationneededtopassproficiencytestsaspartoftherequirementstobecomeaQAfortheOregonExtendedAssessmentsandwasdeliveredbyQTsthroughoutthestate.Thetrainingpackageaddressesthefollowingtopics:
• What'snewin2016-17• 2017TestWindow• Eligibility–whichstudentstakeAA-AAAS?• Testadministration• StudentConfidentiality&TestSecurity• TestAdministration(Physical&Logistic)• Scoring&DataEntry• Reports&SharingResultswithParents• NavigatingtheTrainingandProficiencywebsite• Resources
Appendix2.3B.4
Appendix2.3B.4isthetestcalendarfortheentireOregonstatewideassessmentprogram,includingtheSBA,OAKS,theORExt,theELPA,theKindergartenAssessment,andtheNAEP.
Appendix2.3B.5Appendix2.3B.5isasampleagendathatODEmakesavailabletoQTsaroundthestatetotraintheirrespectivenewQAsastheyimplementthetrain-the-trainersmodelusedbytheOregonExtendedassessment.
Appendix2.3B.6Appendix2.3B.6isthelistofinstructionsprovidedtonewQAsandQTsregardinghowtoaccesstheonlinetrainingandproficiencywebsite.
Appendix2.3B.7Appendix2.3B.7isthelistofresponsibilitiesassociatedwithbeingaQTfortheORExtassessment.
Appendix2.3B.8Appendix2.3B.8isthedocumentthatcontainsthemostcommonlyfieldedquestionsandanswersfromstakeholders,includingparentsandteachers.
Appendix2.3B.9Appendix2.3B.9isthereportthatsummarizesallofthetechnicalassistancequestionsgarneredfromthefieldthisyear.Effortsaremadetofindanypatternsthatourteammayusetoimprovetrainingforthefollowingyear.
2016-2017 OR Extended Assessment Appendix Table and Descriptions– Page
143
Appendix2.3B.10Appendix2.3B.10istheconsequentialvalidityreportforthespring2017consequentialvaliditystudyconductedbyBRT.ThereportprovidesdocumentationoftheperceptionsinthefieldrelatedtobothintendedandunintendedacademicandsocialconsequencesoftheORExt.
Appendix2.3CAppendix2.3CistheORExtPilotTabletAdministrationreportforthespring2017tabletadministration,Phase2,studyconductedbyBRT.Thereportprovidestheresearchplan,summariesofresults,andlessonslearnedregardinghowtoapproachstatewideoperationaltabletadministrationplannedfornextyear.
Appendices2.6Appendix2.6istheguidancethatODEhasprovidedtoassessorstowalkthemthroughtheonlinedataentryprocessfortheORExtonthesecureODEDistrictSecurewebsite.
Appendix2.6CAppendix2.6CisthemanualdefiningthestateofOregon'spoliciesandproceduresregardinghowstudentsareincludedinAMOreporting,includinghowachievement,growth,andgraduationratesarereportedforstudentgroupsandsubgroups.
Appendix3.1AAppendix3.1AisadocumentthatsummarizestheindependentalignmentstudyprocessandparticipantsusedtoreviewthelinkagebetweentheEssentializedStandardsandgradelevelcontentstandards(CCSSinELAandMath;ORSciandNGSSinScience),aswellasthealignmentbetweentestitemsfortheORExtwiththoseEssentializedStandards.Inaddition,reviewersratedtheitemsforpotentialbiasandaccessconcerns.AlldatawasgatheredusingtheDistributedItemReview(DIR)website,supportedbyawebinartrainingandongoingtechnicalassistance.Theresultsofthe2014-15LinkageStudy,whichwasnotindependentbutrunbyBRTresearchers,arealsoincluded.
Appendix3.1BAppendix3.1BisadocumentthatdescribestheDistributedItemReview(DIR)websiteusedbyOregonteacherstoevaluatethealignmentbetweentestitemsfortheORExtwithEssentializedStandards.Inaddition,reviewersratedtheitemsforpotentialbiasandaccessconcerns.AlldatawasgatheredusingtheDIRwebsite,supportedbyawebinartrainingandongoingtechnicalassistance.
Appendix4.1BAppendix4.1BconveysthehistoricaldevelopmentoftheORExtfrom1999tothepresent,includingthegradelevels/bandsassessed,contentareasassessed,andthetargetedcontentstandards.
2016-2017 OR Extended Assessment Appendix Table and Descriptions– Page
144
Appendix4.2Appendix4.2includesthemostcurrentpublishedstateleveldataregardingOregon’sethnicdiversity.
Appendix5.1BAppendix5.1BistherevisedandrigorousguidancethatODEhasprovidedtoIEPteamstoassisttheminmakingappropriateassessmenteligibilitydeterminationsforstudentswithdisabilities.
Appendix5.1DAppendix5.1DincludesasummaryreportofthestatewideresultsandtheadministrationandscoringinstructionsforthenewOregonObservationalRatingAssessment(ORora).TheORoraisadministeredtoallstudentswhoseORExttestingwasdiscontinued.ItprovidesinformationregardingstudentprogressintermsoffunctionalskillsinadaptiveandcommunicationdomainsforthesmallsubgroupofstudentswhoareunabletomeettheacademicexpectationsintheORExt.
Appendix6.1A.1Appendix6.1A.1istheagendaandminutesthatdocumentthehearingandadoptionoftheAAASfortheORExtonJune25,2015.
Appendix6.1A.2
Appendix6.1A.2includesalloftheachievementleveldescriptors(ALDs)andcutscoresthatdefineperformancefortheORExtinqualitativeandquantitativefashions,respectively.TheseAlternateAcademicAchievementStandards(AAAS)describewhatstudentsshouldknowandbeabletodobasedupontheirperformanceontheORExt.
Appendix6.2.1
Appendix6.2.1isthePPTslidesusedtotrainstandardsettersduringtheJune2015standardsettingmeetingsforELA,math,andscience.
Appendix6.2.2Appendix6.2.2isastandardsettingreportgeneratedbyanindependentauditor.ThereportprovidesacomprehensiveevaluationofthebookmarkstandardsettingprocedureemployedfortheORExtonJune15-17,2015.
Appendix6.4CAppendix6.4Cisadocumentthatdisplaystheindividualstudentreport(ISR)thatODEpublishesforstudentswhoparticipateintheORExt.Themock-upincludescutscoresandachievementleveldescriptors(ALDs),aswellaslinkstotheODEwebsiteforadditionalinformation.
Recommended