Oregon Department of Education 2016–2017 Technical Report€¦ · Oregon’s Alternate Assessment System Technical Report: Peer Review Documentation: Critical Elements 1-6 It is

OregonDepartmentofEducation2016–2017TechnicalReportOregon’sAlternateAssessmentSystem

PeerReviewDocumentation:CriticalElements1-6

Oregon’sAlternateAssessmentSystemTechnicalReport:PeerReviewDocumentation:CriticalElements1-6

ItisthepolicyoftheStateBoardofEducationandapriorityoftheOregonDepartmentofEducationthattherewillbenodiscriminationorharassmentonthegroundsofrace,color,religion,sex,sexualorientation,nationalorigin,ageordisabilityinanyeducationalprograms,activitiesoremployment.PersonshavingquestionsaboutequalopportunityandnondiscriminationshouldcontacttheDeputySuperintendentofPublicInstructionwiththeOregonDepartmentofEducation. This technical report is one of a series that describes the development of Oregon’s Statewide Assessment System. The complete set of volumes provides comprehensive documentation of the development, procedures, technical adequacy, and results of the system.

2016-17ORExtendedAssessment PeerReviewCriticalElements–Page

3

TABLEOFCONTENTS

PEERREVIEWCRITICALELEMENTSREFERENCETABLES.....................................................................4OVERVIEW...........................................................................................................................................................14CRITICALELEMENT1:STATEWIDESYSTEMOFSTANDARDSANDASSESSMENTS....................161.1STATEADOPTIONOFACADEMICCONTENTSTANDARDSFORALLSTUDENTS....................................................161.2COHERENTANDRIGOROUSACADEMICCONTENTSTANDARDS..............................................................................221.3REQUIREDASSESSMENTS..............................................................................................................................................231.4POLICIESFORINCLUDINGALLSTUDENTSINASSESSMENTS..................................................................................231.5PARTICIPATIONDATA....................................................................................................................................................24

CRITICALELEMENT2–ASSESSMENTSYSTEMOPERATIONS............................................................262.1TESTDESIGNANDDEVELOPMENT..............................................................................................................................262.2ITEMDEVELOPMENT......................................................................................................................................................272.3TESTADMINISTRATION.................................................................................................................................................282.4MONITORINGTESTADMINISTRATION........................................................................................................................402.5TESTSECURITY................................................................................................................................................................422.6SYSTEMSFORPROTECTINGDATAINTEGRITYANDPRIVACY.................................................................................43

CRITICALELEMENT3–TECHNICALQUALITY:VALIDITY...................................................................463.1OVERALLVALIDITY,INCLUDINGVALIDITYBASEDONCONTENT..........................................................................463.2VALIDITYBASEDONCOGNITIVEPROCESSES.............................................................................................................513.3VALIDITYBASEDONINTERNALSTRUCTURE(CONTENTANDFUNCTION)..........................................................52ANNUALMEASUREABLEOBJECTIVESFREQUENCIES&PERCENTAGES.......................................................................643.4VALIDITYBASEDONRELATIONSTOOTHERVARIABLES........................................................................................72

CRITICALELEMENT4-TECHNICALQUALITY:OTHER.........................................................................774.1RELIABILITY.....................................................................................................................................................................77TESTINFORMATIONFUNCTIONS.........................................................................................................................................784.2FAIRNESSANDACCESSIBILITY......................................................................................................................................904.3FULLPERFORMANCECONTINUUM...........................................................................................................................1134.4SCORING.........................................................................................................................................................................1234.5MULTIPLEASSESSMENTFORMS................................................................................................................................1234.6MULTIPLEVERSIONSOFANASSESSMENT..............................................................................................................1234.7TECHNICALANALYSESANDONGOINGMAINTENANCE.........................................................................................124

CRITICALELEMENT5–INCLUSIONOFALLSTUDENTS.....................................................................1255.1PROCEDURESFORINCLUDINGSWDS......................................................................................................................1255.2A–5.2CPROCEDURESFORINCLUDINGELS...........................................................................................................1275.3ACCOMMODATIONS......................................................................................................................................................1275.4A–5.4EMONITORINGTESTADMINISTRATIONFORSPECIALPOPULATIONS..................................................128

CRITICALELEMENT6–ACADEMICACHIEVEMENTSTANDARDSANDREPORTING................1296.1STATEADOPTIONOFALTERNATEACADEMICACHIEVEMENTSTANDARDSFORSWSCD.............................1296.2ACHIEVEMENTSTANDARDSETTING........................................................................................................................1306.3CHALLENGINGANDALIGNEDACADEMICACHIEVEMENTSTANDARDS..............................................................1316.4REPORTING....................................................................................................................................................................133

CONCLUSIONSANDNEXTSTEPS................................................................................................................134APPENDIXTABLEANDDESCRIPTIONS...................................................................................................137


4

PeerReviewCriticalElementsReferenceTables

CriticalElementsCriticalElement1 StatewidesystemofstandardsandassessmentsCriticalElement2 AssessmentsystemoperationsCriticalElement3 Technicalquality-validityCriticalElement4 Technicalquality-otherCriticalElement5 InclusionofallstudentsCriticalElement6 Academicachievementstandardsandreporting

CriticalElements

CriticalElement1-Statewidesystemofstandardsandassessments1.1Stateadoptionofacademiccontentstandardsforallstudents

TheStateformallyadoptedchallengingacademiccontentstandardsforallstudentsinreading/languagearts,mathematicsandscienceandappliesitsacademiccontentstandardstoallpublicelementaryandsecondaryschoolsandstudentsintheState.

1.2Coherentandrigorousacademiccontentstandards

TheState’sacademiccontentstandardsinreading/languagearts,mathematicsandsciencespecifywhatstudentsareexpectedtoknowandbeabletodobythetimetheygraduatefromhighschooltosucceedincollegeandtheworkforce;containcontentthatiscoherent(e.g.,withinandacrossgrades)andrigorous;encouragetheteachingofadvancedskills;andweredevelopedwithbroadstakeholderinvolvement.

1.3RequiredAssessments

TheState’sassessmentsystemincludesannualgeneralandalternateassessments(basedongrade-levelacademicachievementstandardsoralternateacademicachievementstandards)in:Reading/languageartsandmathematicsineachofgrades3-8andatleastonceinhighschool(grades10-12);Scienceatleastonceineachofthreegradespans(3-5,6-9and10-12).


5

1.4Policiesforincludingallstudentsinassessments

TheStaterequirestheinclusionofallpublicelementaryandsecondaryschoolstudentsinitsassessmentsystemandclearlyandconsistentlycommunicatesthisrequirementtodistrictsandschools.Forstudentswithdisabilities,policiesstatethatallstudentswithdisabilitiesintheState,includingstudentswithdisabilitiespubliclyplacedinprivateschoolsasameansofprovidingspecialeducationandrelatedservices,mustbeincludedintheassessmentsystem;ForEnglishLearners:

A)PoliciesstatethatallEnglishlearnersmustbeincludedintheassessmentsystem,unlesstheStateexemptsastudentwhohasattendedschoolsintheU.S.forlessthan12monthsfromoneadministrationofitsreading/languageartsassessment;

B) IftheStateadministersnativelanguageassessments,theStaterequiresEnglishlearnerstobeassessedinreading/languageartsinEnglishiftheyhavebeenenrolledinU.S.schoolsforthreeormoreconsecutiveyears,exceptifadistrictdetermines,onacase-by-casebasis,thatnativelanguageassessmentswouldyieldmoreaccurateandreliableinformation,thedistrictmayassessastudentwithnativelanguageassessmentsforaperiodnottoexceedtwoadditionalconsecutiveyears.

1.5ParticipationData

TheState’sparticipationdatashowthatallstudents,disaggregatedbystudentgroupandassessmenttype,areincludedintheState’sassessmentsystem.Inaddition,iftheStateadministersend-of-courseassessmentsforhighschoolstudents,theStatehasproceduresinplaceforensuringthateachstudentistestedandcountedinthecalculationofparticipationratesoneachrequiredassessmentandprovidesthecorrespondingdata.


6

CriticalElement2-Assessmentsystemoperations2.1TestDesignandDevelopment

TheState’stestdesignandtestdevelopmentprocessiswell-suitedforthecontent,istechnicallysound,alignstheassessmentstothefullrangeoftheState’sacademiccontentstandards,andincludes:

A) Statement(s)ofthepurposesoftheassessmentsandtheintendedinterpretationsandusesofresults;

B) Testblueprintsthatdescribethestructureofeachassessmentinsufficientdetailtosupportthedevelopmentofassessmentsthataretechnicallysound,measurethefullrangeoftheState’sgrade-levelacademiccontentstandards,andsupporttheintendedinterpretationsandusesoftheresults;

C) ProcessestoensurethateachassessmentistailoredtotheknowledgeandskillsincludedintheState’sacademiccontentstandards,reflectsappropriateinclusionofchallengingcontent,andrequirescomplexdemonstrationsorapplicationsofknowledgeandskills(i.e.,higher-orderthinkingskills);

D) IftheStateadministerscomputer-adaptiveassessments,theitempoolanditemselectionproceduresadequatelysupportthetestdesign.

2.2ItemDevelopment

TheStateusesreasonableandtechnicallysoundprocedurestodevelopandselectitemstoassessstudentachievementbasedontheState’sacademiccontentstandardsintermsofcontentandcognitiveprocess,includinghigher-orderthinkingskills.

2.3TestAdministration

TheStateimplementspoliciesandproceduresforstandardizedtestadministration,specificallytheState:

A) Hasestablishedandcommunicatestoeducatorsclear,thoroughandconsistentstandardizedproceduresfortheadministrationofitsassessments,includingadministrationwithaccommodations;

B) HasestablishedprocedurestoensurethatallindividualsresponsibleforadministeringtheState’sgeneralandalternateassessmentsreceivetrainingontheState’sestablishedproceduresfortheadministrationofitsassessments;

C) IftheStateadministerstechnology-basedassessments,theStatehasdefinedtechnologyandotherrelatedrequirements,includedtechnology-basedtestadministrationinitsstandardizedproceduresfortestadministration,andestablishedcontingencyplanstoaddresspossibletechnologychallengesduringtestadministration.

2.4Monitoringtestadministration

TheStateadequatelymonitorstheadministrationofitsStateassessmentstoensurethatstandardizedtestadministrationproceduresareimplementedwithfidelityacrossdistrictsandschools.


7

2.5TestSecurity TheStatehasimplementedanddocumentedanappropriatesetofpoliciesandprocedurestopreventtestirregularitiesandensuretheintegrityoftestresultsthrough:

A) Preventionofanyassessmentirregularities,includingmaintainingthesecurityoftestmaterials,propertestpreparationguidelinesandadministrationprocedures,incident-reportingprocedures,consequencesforconfirmedviolationsoftestsecurity,andrequirementsforannualtrainingatthedistrictandschoollevelsforallindividualsinvolvedintestadministration;

B) Detectionoftestirregularities;C) Remediationfollowinganytestsecurityincidentsinvolvingany

oftheState’sassessments;D) Investigationofallegedorfactualtestirregularities.

2.6Systemsforprotectingdataintegrityandprivacy

TheStatehaspoliciesandproceduresinplacetoprotecttheintegrityandconfidentialityofitstestmaterials,test-relateddata,andpersonallyidentifiableinformation,specifically:

A) Toprotecttheintegrityofitstestmaterialsandrelateddataintestdevelopment,administration,andstorageanduseofresults;

B) Tosecurestudent-levelassessmentdataandprotectstudentprivacyandconfidentiality,includingguidelinesfordistrictsandschools;

C) Toprotectpersonallyidentifiableinformationaboutanyindividualstudentinreporting,includingdefiningtheminimumnumberofstudentsnecessarytoallowreportingofscoresforallstudentsandstudentgroups.

CriticalElement3-Technicalquality-validity3.1Overallvalidity,includingvaliditybasedoncontent

TheStatehasdocumentedadequateoverallvalidityevidenceforitsassessments,andtheState’svalidityevidenceincludesevidencethattheState’sassessmentsmeasuretheknowledgeandskillsspecifiedintheState’sacademiccontentstandards,including:

A) DocumentationofadequatealignmentbetweentheState’sassessmentsandtheacademiccontentstandardstheassessmentsaredesignedtomeasureintermsofcontent(i.e.,knowledgeandprocess),thefullrangeoftheState’sacademiccontentstandards,balanceofcontent,andcognitivecomplexity;

B) IftheStateadministersalternateassessmentsbasedonalternateacademicachievementstandards,theassessmentsshowadequatelinkagetotheState’sacademiccontentstandardsintermsofcontentmatch(i.e.,nounrelatedcontent)andthebreadthofcontentandcognitivecomplexitydeterminedintestdesigntobeappropriateforstudentswiththemostsignificantcognitivedisabilities.


8

3.2Validitybasedoncognitiveprocesses

TheStatehasdocumentedadequatevalidityevidencethatitsassessmentstaptheintendedcognitiveprocessesappropriateforeachgradelevelasrepresentedintheState’sacademiccontentstandards.

3.3Validitybasedoninternalstructure

TheStatehasdocumentedadequatevalidityevidencethatthescoringandreportingstructuresofitsassessmentsareconsistentwiththesub-domainstructuresoftheState’sacademiccontentstandardsonwhichtheintendedinterpretationsandusesofresultsarebased.

3.4Validitybasedonrelationstoothervariables

TheStatehasdocumentedadequatevalidityevidencethattheState’sassessmentscoresarerelatedasexpectedwithothervariables.

CriticalElement4-Technicalquality-other4.1Reliability TheStatehasdocumentedadequatereliabilityevidenceforits

assessmentsforthefollowingmeasuresofreliabilityfortheState'sstudentpopulationoverallandeachstudentgroupand,iftheState'sassessmentsareimplementedinmultipleStates,fortheassessmentoverallandeachstudentgroup,including:

A) TestreliabilityoftheState'sassessmentsestimatedforitsstudentpopulation;

B) OverallandconditionalstandarderrorofmeasurementoftheState'sassessments;

C) Consistencyandaccuracyofestimatesincategoricalclassificationdecisionsforthecutscoresandachievementlevelsbasedontheassessmentresults;

Forcomputer-adaptivetests,evidencethattheassessmentsproducetestformswithadequatelypreciseestimatesofastudent'sachievement.

4.2Fairnessandaccessibility

TheStatehastakenreasonableandappropriatestepstoensurethatitsassessmentsareaccessibletoallstudentsandfairacrossstudentgroupsinthedesign,developmentandanalysisofitsassessments.

4.3Fullperformancecontinuum

TheStatehasensuredthateachassessmentprovidesanadequatelypreciseestimateofstudentperformanceacrossthefullperformancecontinuum,includingforhigh-andlow-achievingstudents.

4.4Scoring TheStatehasestablishedanddocumentedstandardizedscoringproceduresandprotocolsforitsassessmentsthataredesignedtoproducereliableresults,facilitatevalidscoreinterpretations,andreportassessmentresultsintermsoftheState’sacademicachievementstandards.


9

4.5Multipleassessmentforms

IftheStateadministersmultipleformswithinacontentareaandgradelevel,withinoracrossschoolyears,theStateensuresthatallformsadequatelyrepresenttheState’sacademiccontentstandardsandyieldconsistentscoreinterpretationssuchthattheformsarecomparablewithinandacrossschoolyears.

4.6Multipleversionsofanassessment

IftheStateadministersassessmentsinmultipleversionswithinacontentarea,gradelevel,orschoolyear,theState:

A) Followedadesignanddevelopmentprocesstosupportcomparableinterpretationsofresultsforstudentstestedacrosstheversionsoftheassessments;

B) Documentedadequateevidenceofcomparabilityofthemeaningandinterpretationsoftheassessmentresults.

4.7Technicalanalysesandongoingmaintenance

TheStatehasasystemformonitoringandmaintaining,andimprovingasneeded,thequalityofitsassessmentsystem,includingclearandtechnicallysoundcriteriafortheanalysesofalloftheassessmentsinitsassessmentsystem(i.e.,generalassessmentsandalternateassessments).


10

CriticalElement5-Inclusionofallstudents5.1ProceduresforincludingSWDs

TheStatehasinplaceprocedurestoensuretheinclusionofallpublicelementaryandsecondaryschoolstudentswithdisabilitiesintheState’sassessmentsystem,including,ataminimum,guidanceforIEPTeamstoinformdecisionsaboutstudentassessmentsthat:

A) Providesclearexplanationsofthedifferencesbetweenassessmentsbasedongrade-levelacademicachievementstandardsandassessmentsbasedonalternateacademicachievementstandards,includinganyeffectsofStateandlocalpoliciesonastudent’seducationresultingfromtakinganalternateassessmentbasedonalternateacademicachievementstandards;

B) Statesthatdecisionsabouthowtoassessstudentswithdisabilitiesmustbemadebyastudent’sIEPTeambasedoneachstudent’sindividualneeds;

C) Providesguidelinesfordeterminingwhethertoassessastudentonthegeneralassessmentwithoutaccommodation(s),thegeneralassessmentwithaccommodation(s),oranalternateassessment;

D) Providesinformationonaccessibilitytoolsandfeaturesavailabletostudentsingeneralandassessmentaccommodationsavailableforstudentswithdisabilities;

E) Providesguidanceregardingselectionofappropriateaccommodationsforstudentswithdisabilities;

F) IncludesinstructionsthatstudentseligibletobeassessedbasedonalternateacademicachievementstandardsmaybefromanyofthedisabilitycategorieslistedintheIDEA;

G) Ensuresthatparentsofstudentswiththemostsignificantcognitivedisabilitiesareinformedthattheirstudent’sachievementwillbebasedonalternateacademicachievementstandardsandofanypossibleconsequencesoftakingthealternateassessmentsresultingfromdistrictorStatepolicy(e.g.,ineligibilityforaregularhighschooldiplomaifthestudentdoesnotdemonstrateproficiencyinthecontentareaontheState’sgeneralassessments);

H) TheStatehasproceduresinplacetoensurethatitsimplementationofalternateacademicachievementstandardsforstudentswiththemostsignificantcognitivedisabilitiespromotesstudentaccesstothegeneralcurriculum.


11

5.2ProceduresforincludingELs

TheStatehasinplaceprocedurestoensuretheinclusionofallEnglishlearnersinpublicelementaryandsecondaryschoolsintheState’sassessmentsystemandclearlycommunicatesthisinformationtodistricts,schools,teachers,andparents,including,ataminimum:

A) ProceduresfordeterminingwhetheranEnglishlearnershouldbeassessedwithaccommodation(s);

B) InformationonaccessibilitytoolsandfeaturesavailabletoallstudentsandassessmentaccommodationsavailableforEnglishlearners;

C) GuidanceregardingselectionofappropriateaccommodationsforEnglishlearners.

5.3Accommodations

TheStatemakesavailableappropriateaccommodationsandensuresthatitsassessmentsareaccessibletostudentswithdisabilitiesandEnglishlearners.Specifically,theState:

A) EnsuresthatappropriateaccommodationsareavailableforstudentswithdisabilitiesunderIDEAandstudentscoveredbySection504;

B) EnsuresthatappropriateaccommodationsareavailableforEnglishlearners;

C) Hasdeterminedthattheaccommodationsitprovides(i)areappropriateandeffectiveformeetingtheindividualstudent’sneed(s)toparticipateintheassessments,(ii)donotaltertheconstructbeingassessed,and(iii)allowmeaningfulinterpretationsofresultsandcomparisonofscoresforstudentswhoneedandreceiveaccommodationsandstudentswhodonotneedanddonotreceiveaccommodations;

D) Hasaprocesstoindividuallyreviewandallowexceptionalrequestsforasmallnumberofstudentswhorequireaccommodationsbeyondthoseroutinelyallowed.

5.4Monitoringtestadministrationforspecialpopulations

TheStatemonitorstestadministrationinitsdistrictsandschoolstoensurethatappropriateassessments,withorwithoutappropriateaccommodations,areselectedforstudentswithdisabilitiesunderIDEA,studentscoveredbySection504,andEnglishlearnerssothattheyareappropriatelyincludedinassessmentsandreceiveaccommodationsthatare:

A) ConsistentwiththeState’spoliciesforaccommodations;B) Appropriateforaddressingastudent’sdisabilityorlanguage

needsforeachassessmentadministered;C) Consistentwithaccommodationsprovidedtothestudents

duringinstructionand/orpractice;D) Consistentwiththeassessmentaccommodationsidentifiedbya

student’sIEPTeamor504teamforstudentswithdisabilities,oranotherprocessforanEnglishlearner;

E) Administeredwithfidelitytotestadministrationprocedures.


12

CriticalElement6-Academicachievementstandardsandreporting6.1Stateadoptionofacademicachievementstandardsforallstudents

TheStateformallyadoptedchallengingacademicachievementstandardsinreading/languagearts,mathematicsandinscienceforallstudents,specifically:

A) TheStateformallyadoptedacademicachievementstandardsintherequiredtestedgradesand,atitsoption,alsoalternateacademicachievementstandardsforstudentswiththemostsignificantcognitivedisabilities;

B) TheStateappliesitsgrade-levelacademicachievementstandardstoallpublicelementaryandsecondaryschoolstudentsenrolledinthegradetowhichtheyapply,withtheexceptionofstudentswiththemostsignificantcognitivedisabilitiestowhomalternateacademicachievementstandardsmayapply;

C) TheState’sacademicachievementstandardsand,asapplicable,alternateacademicachievementstandards,include:(a)Atleastthreelevelsofachievement,withtwoforhighachievementandathirdforlowerachievement;(b)descriptionsofthecompetenciesassociatedwitheachachievementlevel;and(c)achievementscoresthatdifferentiateamongtheachievementlevels.

6.2Achievementstandardsetting

TheStateusedatechnicallysoundmethodandprocessthatinvolvedpanelistswithappropriateexperienceandexpertiseforsettingitsacademicachievementstandardsandalternateacademicachievementstandardstoensuretheyarevalidandreliable.

6.3Challengingandalignedacademicachievementstandards

TheState’sacademicachievementstandardsarechallengingandalignedwiththeState’sacademiccontentstandardssuchthatahighschoolstudentwhoscoresattheproficientorabovelevelhasmasteredwhatstudentsareexpectedtoknowandbeabletodobythetimetheygraduatefromhighschoolinordertosucceedincollegeandtheworkforce.IftheStatehasdefinedalternateacademicachievementstandardsforstudentswiththemostsignificantcognitivedisabilities,thealternateacademicachievementstandardsarelinkedtotheState’sgrade-levelacademiccontentstandardsorextendedacademiccontentstandards,showlinkagetodifferentcontentacrossgrades,andreflectprofessionaljudgmentofthehighestachievementstandardspossibleforstudentswiththemostsignificantcognitivedisabilities.


13

6.4Reporting TheStatereportsitsassessmentresults,andthereportingfacilitatestimely,appropriate,credible,anddefensibleinterpretationsandusesofresultsforstudentstestedbyparents,educators,Stateofficials,policymakersandotherstakeholders,andthepublic,including:

A) TheStatereportstothepublicitsassessmentresultsonstudentachievementateachproficiencylevelandthepercentageofstudentsnottestedforallstudentsandeachstudentgroupaftereachtestadministration;

B) TheStatereportsassessmentresults,includingitemizedscoreanalyses,todistrictsandschoolssothatparents,teachers,principals,andadministratorscaninterprettheresultsandaddressthespecificacademicneedsofstudents,andtheStatealsoprovidesinterpretiveguidestosupportappropriateusesoftheassessmentresults;

C) TheStateprovidesfortheproductionanddeliveryofindividualstudentinterpretive,descriptive,anddiagnosticreportsaftereachadministrationofitsassessmentsthat:1) Providevalidandreliableinformationregardinga

student’sachievement;2) Reportthestudent’sachievementintermsoftheState’s

grade-levelacademicachievementstandards(includingperformance-leveldescriptors);

3) Provideinformationtohelpparents,teachers,andprincipalsinterpretthetestresultsandaddressthespecificacademicneedsofstudents;

4) Areavailableinalternateformats(e.g.,Brailleorlargeprint)uponrequestand,totheextentpracticable,inanativelanguagethatparentscanunderstand;

5) TheStatefollowsaprocessandtimelinefordeliveringindividualstudentreportstoparents,teachers,andprincipalsassoonaspracticableaftereachtestadministration.

2016-2017ORExtendedAssessment Overview&CriticalElement1–Page14

OverviewThisdocumentprovidesupdatedtechnicaladequacydocumentationfortheOregonExtendedAssessment(ORExt),whichisOregon'salternateassessmentbasedonalternateacademicachievementstandards(AA-AAAS).Thedocumentationincludestestdesignanddevelopment,technicalcharacteristicsoftheassessments,andtheiruses,andimpactinprovidingproficiencydataongradelevelstatestandardsaspartofthemandatesfromtheEveryStudentSucceedsActof2015(ESSA).TheORExtassessmentswereredesignedin2014-15,includingaverticalscaleinGrades3-8inEnglishlanguageartsandmathematicstosupporteventualdeterminationsofstudentgrowthovertime.ThetestisalignedtoEssentializedStandards(EsSt)thatarepartofcomprehensiveEssentializedAssessmentFrameworks(EAFs)thatwerewrittenatthreelevelsofcomplexity(low,medium,andhigh).TheEsSthavebeenlinkedtogradelevelcontentandexpectations,butsystematicallyreducedintermsofdepth,breadth,andcomplexity(RDBC).AllORExtitemsemployedinthe2016-17ORExtadministrationweredevelopedin2014-15.AnalignmentstudywasconductedatthattimeanditwasdeterminedthatallitemswerealignedtothenewEsSt.AstatewidesampleofOregongeneralandspecialeducationteachershavereviewedalltestitemsfor:1)alignmenttotheEAFs,2)accessibilityforstudentswithsignificantcognitivedisabilities,3)sensitivity,and4)bias.Alloperationalitemsmettheestablishedcriteria.Inaddition,AchievementLevelDescriptors(ALDs)werealsoreviewedforalignmenttotheEsSt.SeeSections1.1,1.2,6.1,and6.3foradditionalinformationrelatedtothecomprehensivegradelevelstandardstoEsStlinkage,aswellasalignmentofitemstotheEsSt.TheORExttestdesignsupportsstudentaccess,includingaccesstoreadaloudfordirectionsandprompts,presentationofoneitemperpage,anditemsdesignedatthreelevelsofcomplexitywherethelowlevelcomplexityitemsincludegraphicand/orobjectsupport.Forassessors,thescoringprocesshasalsobeensimplified,withanswersbeingeithercorrect(1)orincorrect(0).PartialcreditisnolongerpartofthescoringmetricfortheORExt.Inaddition,theoneitemperpageformatnotonlyincreasesstudentabilitytofocusattention,butalsoreducestheburdenonassessorstomaskitemsthatarenotbeingtested.Thefieldappearstohavebeenappreciativeoftheredesign,particularlytheEssentializedStandardsandnewaccessandefficiencyfeatures.Inadditiontodevelopingandreviewing/editingover5,000newitems,conductinganoperationalfieldtest,anddevelopingaverticalscale,thedevelopmentofanewORExtrequiredthatnewAlternateAcademicAchievementStandards(AAAAS)bedevelopedandapproved.ComprehensiveStandardSettingmeetingswereconductedonJune15-17,2015,whichwerethenapprovedbytheOregonStateBoardofEducationonJune25,2015,includingnewachievementleveldescriptors(ALDs)andcutscoresfortheassessments.ComprehensiveAnnualMeasureableObjective(AMO)reportswerefinalizedonJuly10,2015.


Thoughanalignmentstudywasconductedinthefallof2014asdescribedabove,Non-RegulatoryGuidancefromtheU.S.DepartmentofEducation,publishedonSeptember25,2015,includedanexpectationthatallalignmentstudiesmustbeindependent(seeCriticalElement3.1).Anindependentcontractor,Dr.DiannaCarrizales,wasthereforehiredtoperformanadditionalalignmentstudyinthespringof2017.Completeresultsarepresentedinthistechnicalreport(seeSection3.1A-B&3.2).Inadditiontotheindependentalignmentstudy,apilottabletadministrationstudywasalsoconductedinthe2016-17schoolyear.Thisisthesecondphaseofathree-yearplantomaketabletadministrationoftheORExtavailablebythe2017-18schoolyear.AsummaryofthepilottabletadministrationstudyisprovidedinSection2.3C.Completeresultsfromthepilottabletadministrationstudy,phasetwo,arepresentedinAppendix2.3C.Theindependentalignmentstudyandpilottabletadministrationstudyarepartofourfive-yeartechnicaldocumentationplan.Futureplansincludeaninter-raterreliabilitystudy,aswellasanalysesoftheimpactofaccommodations.


CriticalElement1:StatewideSystemofStandardsandAssessments1.1StateAdoptionofAcademicContentStandardsforAllStudentsTheOregonStateBoardofEducation(SBE)adoptednew,challengingacademiccontentstandards,theCommonCoreStateStandards(CCSS),inEnglishlanguageartsandmathematicsinGradesK-12onOctober28,2010.TheseCCSSareutilizedforallstudentsinOregon'spublicschools.OregonwasactivelyinvolvedinthedevelopmentoftheCCSS,astheOregonDepartmentofEducation(ODE),theEducationalEnterpriseSteeringCommittee(EESC),Oregon'sEducationServiceDistricts,andschooldistrictrepresentativesprovidedfeedbackonthedraftCCSSstandards.Similarly,theSBEadoptedtheNextGenerationScienceStandards(NGSS)onMarch6,2014.TheNGSSestablishlearningtargetsforallstudentsinOregon'spublicschoolsinGradesK-12.TheODEandtheOregonScienceContentandAssessmentPanelprovideddirectfeedbackrelatedtotheNGSS.TheNGSSarebeingphasedinovertimeinstructionally,sostudentsarebeingassessedrelativetotheOregonScience(ORSci)standardsthatwereadoptedin2009.Forstatecontentstandards,seehttp://www.oregon.gov/ode/educator-resources/standards/Pages/default.aspx.Thetablebelowprovidesexamplesofessentializedstandardsingrades5,8,&11inthesubjectareasofEnglishlanguagearts(ELA),mathematics,andscience.CompleteEAFspreadsheetsareavailableatthelinkprovidedhere,aswell.Intherightcolumnaredesignationsforestimateddifficultyofanitem:L(low),M(medium),andH(high).

http://www.brtprojects.org/publications/training-modulesSeeAppendix1.1foraUserGuidethatexplainsthedevelopmentprocessandintendedusesfortheEAFs.


GRADE5

Area Cluster Standard Sub-StandardEssential-izedStandard

L/M/HDescriptors

ReadingStandardsforLiteratureK–5

1.KeyIdeasandDetails

3.Compareandcontrast2ormorecharacters,settings,oreventsinastoryordrama,drawingonspecificdetailsinthetext(e.g.,howcharactersinteract).

None

Identifyacharacter,setting,oreventinastoryreadtostudent.

L-Sentenceof7wordsorlessthatcontains1character,setting,oreventreadtostudent.M-2shortsentencesthatcontain1character,setting,oreventreadtostudent.H-2mediumsentencesthatcontain1character,setting,oreventreadtostudent.

MathNumber&OperationsinBaseTen

1.Understandtheplacevaluesystem.

1.Recognizethatinamulti-digitnumber,adigitinoneplacerepresents10timesasmuchasitrepresentsintheplacetoitsrightand1/10ofwhatitrepresentsintheplacetoitsleft.

Useplacevaluetocomparenumbersthataremultiplesof10andones'versustens'placeand.5.

L-identifymultiplesof10:10,20,30,40,50,60.M-identifytherelationbetweentheplacevaluesforthedouble-digitnumbers11,22,33,44,55.H-identifywhichnumberisintheten'splaceandone'splace.


Area Cluster Standard Sub-StandardEssential-izedStandard

L/M/HDescriptors

Science*MatterandItsInteractions

NGSSStandard:Measureandgraphquantitiestoprovideevidencethatregardlessofthetypeofchangethatoccurswhenheating,cooling,ormixingsubstances,thetotalweightofmatterisconserved.

ORScienceStandards:5.3S.1Basedonobservationsandscienceprinciples,identifyquestionsthatcanbetested,designanexperimentorinvestigation,andidentifyappropriatetools.Collectandrecordmultipleobservationswhileconductinginvestigationsorexperimentstotestascientificquestionorhypothesis.5.3S.2Identifypatternsindatathatsupportareasonableexplanationfortheresultsofaninvestigationorexperimentandcommunicatefindingsusinggraphs,charts,maps,models,andoralandwrittenreports.

Measureand/orcomparetheweightofdifferenttypesofmatter.

L-Measuretheweight/massofcommonobjectsinvariousphasesofmatterusingpicturesofsuchobjects(i.e.,anobjectonascalethatweighs3pounds);M-Comparetheweight/massofcommonobjectsinvariousstagesofmatterusingpicturesofsuchobjects(e.g.,aballoonweighslessthanarockorglassofwater)-Choosethecorrecttooltomeasuretheweight/massofobjects;H-Comparetheweight/massofcommonobjectsinvariousphasesofmatterusinggraphsanddata.

Note.Thescienceessentializedstandardsaredually-linkedtobothNGSSandOregonSciencestandards,respectively.BothgeneraleducationstandardsarethuslistedforscienceintheseEAFtables.


GRADE8Area Cluster Standard Sub-Standard Essentialized

Standard L/M/HDescriptors

ReadingStandardsforLiterature6–12

2.CraftandStructure

6.Analyzehowdifferencesinthepointsofviewofthecharactersandtheaudienceorreader(e.g.,createdthroughtheuseofdramaticirony)createsucheffectsassuspenseorhumor.

None

Identifythenarratororacharacterinastoryreadtostudent.

L-3sentencesthatcontain2charactersornarratorsreadtostudent.M-Paragraphof4sentencesthatcontains2charactersornarratorsreadtostudent.H-Paragraphof5sentencesthatcontains2charactersornarratorsreadtostudent.

MathStatistics&Probability

1.Investigatepatternsofassociationinbivariatedata.

3.Usetheequationofalinearmodeltosolveproblemsinthecontextofbivariatemeasurementdata,interpretingtheslopeandintercept.Forexample,inalinearmodelforabiologyexperiment,interpretaslopeof1.5cm/hrasmeaningthatanadditionalhourofsunlighteachdayisassociatedwithanadditional1.5cminmatureplantheight.

Compareratesusingslower/less,faster/more,same(mph,beatspersecond,$perhour,$perlb).

L-identifyfasterrateusing(0-20).M-identifyslower,faster,orsamerateusing(21-50).H-identifyslower,faster,orsamerateusing(51-100).

Science Energy

NGSSStandard:Plananinvestigationtodeterminetherelationshipsamongtheenergytransferred,thetypeofmatter,themass,andthechangeintheaveragekineticenergyoftheparticlesasmeasuredbythetemperatureofthesample.

ORScienceStandards8.2P.2Explainhowenergyistransferred,transformed,andconserved.

Recognizetemperatureasameasureofhowhotorcoldmatteris,andthatheatistransferable.

L-Recognizethedifferencebetweenhotandcold(e.g.,objects,outside);M-Recognizethathotandcoldarerelatedtomeasuresoftemperature,includingchangesintemperature;H-Identifyexamplesofheattransfer,andhowsuchtransfermightbeminimized/maximized(e.g.,wearingacoattostaywarm).


GRADE11Area Cluster Standard Sub-

StandardEssentializedStandard L/M/HDescriptors

ReadingStandardsforLiterature6–12

2.CraftandStructure

4.Determinethemeaningofwordsandphrasesastheyareusedinthetext,includingfigurativeandconnotativemeanings;analyzetheimpactofspecificwordchoicesonmeaningandtone,includingwordswithmultiplemeaningsorlanguagethatisparticularlyfresh,engaging,orbeautiful.(IncludeShakespeareaswellasotherauthors.)

None

Identifythemeaningoffigurative,connotative,orwordswith2ormoremeanings.

L-Paragraphof4sentencesreadtostudent.M-Paragraphof5sentencesreadtostudent.H-2paragraphsreadtostudent.

Math

ExpressingGeometricPropertieswithEquations

2.Usecoordinatestoprovesimplegeometrictheoremsalgebraically

7.Usecoordinatestocomputeperimetersofpolygonsandareasoftrianglesandrectangles,e.g.,usingthedistanceformula.

Identifytheperimeteroftriangles,squares,rectangles,andpentagons.

L-identifyperimeteroftriangleswithsidelengths(1-5).M-identifytheperimeterofsquaresandrectangleswithsidelengths(1-10).H-identifytheperimeterofpentagonswithsidelengths(1-20).


Area Cluster Standard Sub-Standard

EssentializedStandard L/M/HDescriptors

Science Earth’sSystems

DevelopamodeltoillustratehowEarth’sinternalandsurfaceprocessesoperateatdifferentspatialandtemporalscalestoformcontinentalandocean-floorfeatures.

H.1E.2DescribethestructureandcompositionofEarth’satmosphere,geosphere,andhydrosphere.H.2E.1Identifyandpredicttheeffectofenergysources,physicalforces,andtransferprocessesthatoccurintheEarthsystem.Describehowmatterandenergyarecycledbetweensystemcomponentsovertime.H.2E.2ExplainhowEarth’satmosphere,geosphere,andhydrospherechangeovertimeandatvaryingrates.ExplaintechniquesusedtoelucidatethehistoryofeventsonEarth.

Identifydifferent(geoscience)processesthatshapetheEarthincludingassociatedEarthfeatures.(S08ESS2.2)

L-Identifyconditionsthatleadtospecifictypesofsurfaceweathering(i.e.,withwater,ice,orwindasvehicle-Whichshowswatererosion?-ariver,pondorvolcano);M-Identifygeoscienceprocessesthatshapelocalgeographicfeatures(e.g.,earthquakes,volcanoes,meteorites/craters-Whichisanexampleofvolcanism?–picturesofavolcano,river,rain);H-ExtendM-levelquestionsbylinkingfeaturestothegeoscienceprocess(e.g.,Whichtypeoferosionprocesslikelyledtothecanyon?-river,rain,wind;Whichfeatureisassociatedwithrecentvolcanism?-island,valley,river).


1.2CoherentandrigorousAcademicContentStandardsTheCCSS,ORSci,andNGSSdefinewhatstudentsinOregonshouldknowandbeabletodobythetimetheygraduatefromhighschool.TheseCCSS,whichweredevelopedbynationalstakeholdersandeducationexperts,havebeendeterminedtobecoherentandrigorousbyresearchersattheFordhamInstitute(seeAppendix1.2).Theywerealsodevelopedwithwidestakeholderinvolvement,particularlyhereinOregon.ThenewORExtislinkeddirectlytothecontentintheCCSSinEnglishlanguagearts(reading,writing,&language)andmathematics.TheORExtisduallylinkedtotheORSciaswellastheNGSS.TheNGSSarewidelyacceptedbymostrelevantscienceinstructionorganizationsasreflectiveofrigorousandcoherentscienceconcepts(seehttp://www.nextgenscience.org/support-scientific-engineering-and-education-communities).ThenewEssentializedAssessmentFrameworks(EAFs)arepubliclyavailableatthelinkonpage16aboveintheEssentializedAssessmentFrameworkssection.AUserGuideisprovidedtoinstructeducatorsregardingtheintendedusesoftheEssentializedStandards(EsSt),includingthedevelopmentofPresentLevelsofAcademicAchievementandFunctionalPerformance(PLAAFP)andIndividualizedEducationProgram(IEP)goalsandobjectives.ThebasicessentializationprocessemployedtogenerateessentializedstandardsandwritealigneditemsfortheORExtisoutlinedbelow.Theprocesscanalsobeusedtosupportthedevelopmentofcurricularandinstructionalmaterials,foundedinresearch-basedpedagogy.


1.3RequiredAssessmentsTheORExtassessmentswereadministeredinthe2016-17schoolyearinELAandmathinGrades3-8andonceinGrade11;scienceisassessedinGrades5,8,&11.ThisassessmentplanmeetstherequirementsforgradelevelassessmentinGrades3-8andonceinhighschool(Grades10-12)forELAandmathematics,whilescienceisassessedonceinthe3-5gradeband,onceinthe6-9gradeband,andonceinthe10-12gradeband:ContentArea Grade

3Grade4

Grade5

Grade6

Grade7

Grade8

Grade11

Englishlanguagearts X X X X X X XMathematics X X X X X X XScience X X X1.4PoliciesforIncludingAllStudentsinAssessmentsOriginally,Oregonstatuterequiredthatallstudentsparticipateinstatewideassessments,withexceptionsallowedfordistrict-approvedparentrequestforassessmentwaivers(parentopt-outrequests)relatedtostudentdisabilityorreligiousbeliefs(seeOregonAdministrativeRule,OAR§581-022-0612):

ExceptionofStudentswithDisabilitiesfromStateAssessmentTesting(1)Forthepurposesofthisrulea"studentwithadisability"isastudentidentifiedundertheIndividualswithDisabilitiesEducationAct,consistentwithOARchapter581,division015,orastudentwithadisabilityunderSection504oftheRehabilitationActof1973.(2)ApublicagencyshallnotexemptastudentwithadisabilityfromparticipationintheOregonStateAssessmentSystemoranydistrictwideassessmentstoaccommodatethestudent'sdisabilityunlesstheparenthasrequestedsuchanexemption.

However,HouseBill2655establishedaStudentBillofRightsonJanuary1,2016,whichpermittedparentsoradultstudentstoannuallyopt-outofOregon'sstatewidesummativeassessments,pursuanttoOAR§581-022-1910.TheGovernorpublishedamemorandumforSuperintendents,Principals,andDistrictTestCoordinatorsrelatedtothechange(seeAppendix1.4.1).Theexpectationthatallstudentsintheassessedgradesparticipate,includingstudentswithdisabilities,iselaboratedclearlyandpervasivelyacrossallguidancedocuments.ForexampleintheOregonTestAdministrationManual(TAM),whereitstatesthat,"Allstudentsenrolledingrades3–8andinhighschoolmusttaketherequiredOregonStatewideAssessmentsofferedattheirenrolledgrade,includingstudentsre-enrolledinthesamegradeasintheprioryear,unlessthestudentreceivesaparent-requestedexemption..."(seeAppendix1.4.2,p.96).


1.4AEnglishLearnersEnglishlearnersareincludedasappropriateinOregon'sstatewideassessmentsystem.(seeAppendix1.4A.1,pp.31-33).TheSmarterBalancedassessmentdirectionsaretranslatedintomultiplelanguagesandavailableviatheOaksportal.OAR581-022-0620(2)requiresODEtoprovidetranslatedOAKSassessmentsforpopulationsatorabove9%ingradesK-12withinthreeyearsaftertheschoolyearinwhichthelanguageexceedsthethreshold(seeAppendix1.4A.2).Inaddition,theaccommodationsavailabletostudentswhoparticipateintheORExtincludetranslationintothenativelanguage,whereappropriate(seeAppendix2.3A1,pp.36-43).

1.4BNativeLanguageAssessmentsTheORExtisnotadministeredinanativelanguageformat,thoughitcanbetranslatedintoastudent'shomelanguage.1.5ParticipationDataOregon'sparticipationdataindicatethatmoststudentsinthetestedgradelevelsareincludedinourassessmentsystem.Thestudentswithdisabilitiessubgroupdidnotmeetminimumparticipationrequirementsin2015-16,themostcurrentdataavailableatthetimeofthisreport,inEnglishlanguageartsormathematics,withratesat92.3%and91.5%,respectively.Seethetablebelowforasummaryofparticipation.Forcompletereports,seehttp://www.oregon.gov/ode/educator-resources/assessment/Pages/Assessment-Group-Reports-for-2014-2015-and-2015-2016.aspx.DocumentationofthisrequirementisprovidedwithintheAnnualPerformanceReport,IndicatorB3,whichissubmittedtotheUnitedStatesDepartmentofEducation's(USED's)OfficeofSpecialEducationPrograms(OSEP).Participationandperformancesummariesareprovidedbelow.Additionalinformationregardingstateperformanceispublishedinthe2015-16StateReportCard(seeAppendix1.5,pages1-11forstudentandteacherdemographicsandpages21-48forassessmentinformation).


2016-2017ORExtendedAssessment CriticalElement2–Page26

CriticalElement2–AssessmentSystemOperations2.1TestDesignandDevelopmentThetestspecificationsdocumentthatdescribesourapproachtoassessmentandtestdesignfortheORExtispublishedinAppendix2.1.Thedocumentincludesourapproachtoreducingthedepth,breadth,andcomplexity(RDBC)ofgradelevelcontentstandards,anoverviewoftheessentializationprocessandEAFdocuments,theplannedtestdesignfortheORExt,testdevelopmentconsiderations,sampletestitems,itemspecifications,anduniversaltools/designatedsupports/accommodations.Nonewitemsweredevelopedin2016-17,sothe2014-15testspecificationsarethemostcurrentavailable.

2.1AORExtPurposeThestatedpurposeoftheORExtistoprovidethestatetechnicallyadequatestudentperformancedatatoascertainproficiencyongradelevelstatecontentstandardsforstudentswithsignificantcognitivedisabilities.Along-termgoaloftheprogramistoalsoprovideinformationregardingannualstudentgrowthrelatedtothesecontentstandardsoverGrades3-8,asmeasuredbyverticallyscaledassessmentsinELAandmathematics.Theresultsoftheassessmentarecurrentlyreportedincomparisontofourperformancelevels:Level1,Level2,Level3,andLevel4.Levels3and4denoteaproficientlevelofperformance,whileLevels1and2denoteperformancethatisnotproficient.BRTandODEdevelopedascaledscoreinterpretationguidetoassiststakeholdersininterpretingthemeaningofthescaledscoresgeneratedbytheORExt,supportedbythestate'sachievementleveldescriptors.ThisguidanceispublishedinAppendix2.1A.

2.1BORExtTestBlueprint

Appendix2.1BincludestheentiretestblueprintfortheORExt,asconveyedbythebalanceofrepresentationacrosscontentareasanddomains.Field-testingisconductedeachyearinordertosupportthecontinuousimprovementoftestfunctioning.However,itemsareselectedtomaintainthisbalanceofrepresentation.Oregonteachersvalidatedthecontentoftheassessment,agreeingwiththestandardsthatwereandwerenotselectedtodeveloptheEssentializedStandardstowhichtheORExttestitemsarealigned.

2.1CTestDevelopmentProcessesThetestdevelopmentprocessimplementedfortheORExtisconveyedinAppendix2.1C,includingstandardselectionandvalidation,itemdevelopment,itemreview,reviewofallOregonteacherfeedbackandupdatingofitems,andscalinganditemselection.TheappendixarticulatestheprocessusedtogeneratethematerialswithcommaseparatedvaluefilesusedtocreateitemtemplatesthatfedintoAdobeInDesign©throughadatamerge.FinaltestpackagesarereviewedforaccuracyandcontentandthendisseminatedviasecurefiletransfertoOregonQualifiedAssessors.

2.1DComputer-AdaptiveConsiderationsTheORExtisnotacomputer-adaptiveinstrument,sotheseconcernsdonotapply.


2.2ItemDevelopmentItemwriterswererecruitedbyODEstaffonMay20,2014usinganexistingQualifiedAssessor/QualifiedTrainerlistservprogram,usingthefollowingtext:ProjectDescription:BehavioralResearchandTeachingattheUniversityofOregonisrecruitingOregonteacherstoparticipateinitemdevelopmentforanewalternateassessment,tobefieldtestedinthespringof2015.Selectedteacherswillbeaskedtodevelop360itemsinEnglishLanguageArts,Mathematics,orScienceoverthecourseofthesummer,frommid-JunethroughendofAugust2014.TheProjectDirectorwillworkwithleaditemdeveloperstoprovidetraining,ongoingreviewandfeedback,andqualityassurance.Allparticipantswillbeexpectedtoprovidedocumentationoftheirqualificationsandsigntestsecurityagreements.Inaddition,allitemdeveloperswillbeexpectedtoparticipateinahalf-dayitemdevelopmenttrainingbaseduponthefollowingschedule:

• ELA-Tuesday,June17,2014from8AMto12PM• Math–Wednesday,June18,2014from8AMto12PM• Science–Thursday,June19,2014from8AMto12PM

MinimumQualifications:AlllicensedOregonpublicschoolteacherswithatleastthreeyearsofteachinginalifeskills/severeneedsprogram(SPED)orageneraleducationclassroom(GEN-ED),respectively,areencouragedtoapply.Preferencewillbegivenforitemwritingexperience,additionalyearsofteachingexperience,andhighereducationdegreestatus.Compensation:Teacherswhoparticipateinthisprocesswillbecompensatedatarateof$20/hrviaprofessionalservicecontracts.Itisanticipatedthatteacherswillproduce4ELAitems/hr,6Scienceitems/hr,and8Mathitems/hr.Assuch,themaximumcontractamountforELAwillbe$1,800,forScience$1,440,andforMath$900.Itemdevelopmentwillfocusprimarilyonwritingthestemand3options,withnoneedtoproducegraphics(ratheruselabelsforaBRTgraphicdesignertoproduce).Needs ContentArea GradeLevel(s)2teachers(2SPED) ELA Elementary(G3-5)2teachers(1GEN-ED;1SPED)

ELA Middle(G6-8)

1teachers(1GEN-ED) ELA High(G11)TotalNumberNeededELA 5

1teachers(1SPED) Math Elementary(G3-5)1teachers(1SPED) Math Middle(G6-8)2teachers(1SPED;1GEN-ED)

Math High(G11)

TotalNumberNeededMath 4


3teachers(2SPED;1GEN-ED)

Science G5,8,&11

TotalNumberNeededScience 3

TotalOregonTeacherItemWritersNeeded 12Contact:Ifyoumeettheabovequalificationsandareinterestedinapplyingtoassistonthisproject,pleasecontactDanFarleyatdfarley@uoregon.eduorat541-346-3133.ThedeadlinetoapplyisJune13,2014.Thankyou.Becausethetimelinerequiredworkoverthesummer,Oregonteacherrecruitmentwaschallenging.BRTresearchersthusperformedanadditionalon-campusrecruitmentwithintheCollegeofEducationusingthesameinformation.Thefinalpoolofitemwritersincluded18itemwriters:sevenOregonteachers(allwithMAdegrees),fivePhDcandidateswithintheCOE,andsixBRTresearchers(fourPhDcandidates,onePhD,andonewithanMA).Itemwritersaveraged11.5yearsofteachingexperience.TheteachersrecruitedallhadpriorexperiencedevelopingitemsfortheORExt,asdidalloftheBRTresearchers.ThefivePhDcandidateswithintheCOEhadnoprioritemdevelopmentexperience.AllitemdevelopmentwasreviewedbyBRTresearchersandtheProjectManager.TheitemdevelopmentprocessfollowediselaboratedinAppendix2.2.1,whichisthePowerPointusedintrainingallOregonitemwriters.Theitemdevelopmentprocesswasstructuredwiththefollowingsteps.Itemwriterswerefirstorientedtothestudentpopulation,asthepoolofitemwritersincludedbothcontentandspecialeducationexperts.TheEssentializationProcessusedtoRDBCgradelevelstandardswasthenmodeledsowriterswouldunderstandhowtheitemalignmenttargets,theEssentializedStandards,weregenerated.Lecture,guidedpractice,andindependentpracticeactivitiesandfollow-updiscussionensuredcomprehensionoftheprocess.BRTstaffdevelopedexemplaritemsforeveryEssentializedStandard,varyingthecomplexityfromLow(L)toMedium(M)toHigh(H)levelsofcomplexitytoconveythedifferentperformanceexpectationsateachlevel.Thebalancedverticalscalingdesignprovidedanoverallform-to-formandgrade-to-gradelevelframeworkforthetestformationprocessonceitemsweredeveloped(seeAppendix2.2.2).SampleitemsareprovidedinAppendix2.2.3forstakeholderreference,demonstratingtheformatandstyleoftypicalitemsontheORExt.2.3TestAdministrationTheORExtassessmentsareadministeredaccordingtotheadministration,scoring,analysis,andreportingcriteriaestablishedintheORExtGeneralAdministrationManual(seeAppendix2.3).ImportantupdatestothetestingprocessaredistributedviatheAssessmentandAccountabilityUpdateslistserve,aswell(seehttp://www.oregon.gov/ode/educator-resources/assessment/Pages/Assessment-and-Accountability-Update.aspx).ODEusesthissystemtocommunicateinformationthatisrelevantforthestatewideassessmentsystem,includingtheORExt.AnnouncementsaresenttothelistservbyemailandarealsopostedtotheODEwebsite.ThestandardizationoftestadministrationissupportedbyacomprehensivetrainingprocessdescribedbelowinSection2.3B.


2.3AAdministrationandAccommodationsThestatehasensuredthatappropriateuniversaltools,designatedsupports,andaccommodationsareavailabletostudentswithdisabilitiesandstudentscoveredbySection504byprovidingguidanceandtechnicalsupportonaccommodations(seeAppendices2.3A.1and2.3A.2).Guidelinesregardinguseoftheaccommodationsforinstructionalpurposesareincludedinthedocument,asallstudentsareexpectedtoreceivetestaccommodationsthatareconsistentwithinstructionalaccommodations.AccommodationsarebuiltintotheflexibilityprovidedbytheORExttestthoughtheyhavenotyetbeenresearchedfortheORExt.However,annualtrainingandproficiencytestingeffortsrelatedtobecomingaqualifiedassessorand/orqualifiedtrainerfortheORExtsupportstandardizeduseofavailableaccommodationsthatarenotalreadypartofthetestdesign.Basedonannualanalyses,resultsdemonstratethatstudentperformancevariesaccordingtotheirabilitiesandnotconstruct-irrelevantfactors,suchassex,race,orethnicity(SeeSection4.2).ThestatehasensuredthatappropriateaccommodationsareavailabletostudentswithlimitedEnglishproficiencybyprovidingguidanceandtechnicalsupportonaccommodations(seeAppendix2.3A.1).Communicationsystemsforthisstudentpopulationarelimited;exposuretomultiplelanguagescanmakeastudent'scommunicationsystemmorecomplex.TheORExtusesuniversaldesignprinciplesandsimplifiedlanguageapproachesinordertoincreaselanguageaccesstotestcontentforallstudents.Inaddition,directionsandpromptsmaybetranslated/interpretedforstudentsintheirnativelanguage.Ananalysisofaccommodatedversusnon-accommodatedadministrationsisneededinordertodemonstratethattheprovisionoflanguageaccommodationsisnotprovidinganyadvantagetostudentswithlimitedEnglishproficiency,noranydisadvantagetootherparticipants.Accommodationsinformationwascollectedthisyearasanoptionfordataentry.Enteringaccommodationsinformationwillberequirednextyear.AnalysesoftheimpactofaccommodationprovisionontheORExtshouldthusbefeasibleafterthespring2018administration.TheOregonExtendedassessmentscanbeadministeredusingbothLargePrintandBraille(contractedandnon-contracted)versions,aswell.OregonhasensuredthattheOregonExtendedassessmentsprovideanappropriatevarietyofaccommodationsforstudentswithdisabilities.Thestatehasprovidedguidanceonaccommodationsinpresentation,response,setting,andtimingintheAccommodationsManual2013-14:HowtoSelect,Administer,andEvaluateAccommodationsforOregon'sStatewideAssessments(seeAppendix2.3A.2).TheOregonExtendedassessmentsarealsodesignedaccordingtouniversaldesignprinciplesandutilizeasimplifiedlanguageapproach(seeAppendix2.3A.3).Inthe2013-2014schoolyear,thestatedevelopedatrainingandproficiencyprogramforsignlanguageinterpretationofitsassessmentsandhasupdatedthesiteannuallysincethattime.Thetrainingprocess(http://lms.brtprojects.org)includedvideosofinterpretersadministeringitemstostudents,materialsthatsupportappropriateadministration(i.e.,


transcriptsandPowerPointslidesthatsupplementthevideoadministrationsandthecurrentODEaccommodationsmanual),andproficiencytestingtosupportstandardizedinterpretationforOregon'sassessments,includingtheORExt.A10-itemproficiencytestwasadministered,withan80%requiredforpassing(8/10itemscorrect).In2016-17,thesitewasusedtotrain60participants.Allparticipantspassedtheassessmentonthefirstattempt.Theoverallaveragescoreontheproficiencytestwas97.6%.TheORExtassessmentsprovideanappropriatevarietyoflinguisticaccommodationsforstudentswithlimitedEnglishproficiency.Theyalsouseasimplifiedlanguageapproachintestdevelopmentinordertoreducelanguageloadofallitemssystematically(seeAppendix2.3A3).Anygivenstudent'scommunicationsystemmayincludehomesigns,schoolsigns,Englishwords,andSpanishwords,forexample.Withtheexceptionofitemsthatrequireindependentreading,theORExtassessmentcanbetranslatedorinterpretedbyaQualifiedAssessor(QA)workingwithaninterpreterinthestudent'snativelanguage,includingAmericanSignLanguage.QAsareallowedtotranslate/interpretthetestdirections.QAscanadapttheassessmenttomeettheneedsofthestudent,whilestillmaintainingstandardizationduetosystematicpromptsandwell-definedanswers.

2.3BComprehensiveTrainingSystemComprehensiveinformationforongoingtrainingforallqualifiedassessors(QAs)andQualifiedTrainers(QTs)isprovidedinAppendices2.3B.1-2.3B.8.TrainingandQA/QTproficiencyisdeterminedannuallyviaanonlinedistributionandassessmentsystemlocatedathttps://or.k12test.com.Thiswebsitehostsallresourcesandinformationneededtoadminister,score,report,andinterprettheresultsfromtheORExt.ThewebsitealsoincludesproficiencyassessmentsthatarerequiredforallQAsandQTswhomayadministertheORExt.QTsaredirectlytrainedbyODEandBRTstaffaspartofatrainthetrainersmodel.QTsthenprovidedirecttrainingsfornewQAsintheirrespectiveregions.TheOregonDepartmentofEducation(ODE)providedfourdirectstatewidetrainingsfornewQualifiedTrainers(QTs)andQualifiedAssessors(QAs)inface-to-faceregionaltrainings.Theschedulefortheregionaltrainings,aswellasrelevanttraininginformation,isprovidedbelow:


Date Who/Team Location

11-1-2016

Team:BradLenhardt,GeraldTindal,&SevrinaTindalContact:[email protected]

IMESDPendleton,OR

11-3-2016

Team:BradLenhardt,GeraldTindal,&SevrinaTindalContact:[email protected]

HDESD-Redmond,OR

11-10-2016Team:BradLenhardt&DanFarleyContact:[email protected]

SOESD-Medford,OR

11-15-2016Team:BradLenhardt&DanFarleyContact:[email protected]

NWESDHillsboro,OR

11-17-2016

Team;BradLenhardt&DanFarleyContact:[email protected] WillametteESD

Salem,OR

OnlytrainedQualifiedAssessors(QAs)canadministertheOregonExtendedassessment.QualifiedAssessorswhoalsoreceivedirectinstructionfromODEandBRTmaybecomeQualifiedTrainers(QTs)whoarecertifiedtotrainlocalstaffusingthetrain-the-trainersmodel.Trainingfornewassessorsmustbecompletedonanannualbasis.Assessorswhodonotmaintaintheirrespectivecertificationsforanygivenyearmustre-trainiftheychoosetoenterthesystemagain.ThetablesbelowcontaindatafromtheOregonExtendedAssessmentTrainingandProficiencyWebsite(http://or.k12test.com/).AllassessorsneedtocompletesomeformoftrainingeachyeartoretaintheirstatusforadministeringtheExtendedAssessments.Newassessorsandreturningassessorswhoneededfurthertrainingin2016-17wererequiredtopassfourproficiencieswithascoreof80%orhigher.ThesefourproficiencieswereinAdministration,EnglishLanguageArts(ELA),Mathematics,andScience.ReturningQAsorQTsforthe2016-17schoolyearonlyneededtopassaRefresherProficiency,againwithascoreof80%orhigher.Thetablesbelowcontaindataonthenumberofassessors(participants)ineachofthefourproficiencies,aswellastheRefresherProficiency.Includedinthedataisthenumberofattemptsneededtoattainapassingscoreaswellastheaveragepassingscoreoftheparticipants.


AnanalysisoftheOregonExtendedAssessmentTrainingandProficiencyWebsiteshowed353Assessorsin-Training,1,030QualifiedAssessors,and137QualifiedTrainers.322TestParticipants–AdministrationProficiency

NumberofParticipants

PercentageofParticipants AttemptstoPass AveragePassing

Score270 83.9% 1 91.5%44 13.7% 2 90%6 1.9% 3 84%1 <1% 4 0%1 <1% 5 85%

268TestParticipants–EnglishLanguageArtsProficiency



Score259 96.6% 1 95.7%9 3.4% 2 87.8%

264TestParticipants–MathematicsProficiency



Score259 98.1% 1 96.6%4 1.5% 2 90%1 <1% 3 100%

259TestParticipants–ScienceProficiency



Score257 99.2% 1 96.9%1 <1% 2 0%1 <1% 3 100%

936TestParticipants–RefresherProficiency



Score914 97.6% 1 93.7%22 2.4% 2 91.6%2 <1% 3 96%

AhighernumberofassessorscompletedtheRefresherProficiencytestthanthesubjectareaproficiencytestsreflectingagreaternumberofreturnassessorscomparedtonewassessors.AdministrationProficiencycontinuedtobethemostchallengingtonewassessors,butmostwereabletopassonthefirstorsecondattemptwithlessthan2%ofassessorsrequiringmorethantwoattempts.ThemajorityofassessorspassedtheELA,


Math,Science,andtheRefresherproficiencytestsonthefirstattemptwithlessthan4%requiringasecondorthirdattempt.Therewere73fewerQualifiedAssessorsbut8moreQualifiedTrainerscomparedtolastyear.EvaluationsarecollectedateachQTtraininginNovember.Theresultsreflectgeneralapproval,butalsosuggestareasofimprovementthatODEandBRTworkonforsubsequenttrainings/subsequentyears,asappropriate.QTevaluationsthisyearincludedpositivelywordedstatementsregardingthequalityoftrainingratedonascalewhere1=StronglyDisagree,2=Disagree,3=Agree,and4=StronglyAgree.Thefirstsectionevaluatedthestate-levelinformationandtheknowledgeoftheODEpresenters,theparticipants’levelofcomfortwiththetrainingprovided,theparticipants’abilitytocarrythistrainingandmaterialsbacktotraindistrictstaff,andtheoverallutilityofthetraining.Seventy-eightpercentofparticipantsstronglyagreedwiththesestatements,19%agreed,andlessthan3%disagreedandstronglydisagreed,collectively.Inthesecondsection,participantswereaskedtoevaluatetheBRTtrainersandtheirguidelinesregardinghowtousethetrainingandproficiencywebsiteandrelatedresources.Seventy-ninepercentofparticipantsstronglyagreedwiththesestatements,19%agreed,andlessthan2%disagreedandstronglydisagreed,collectively.Overall,theseresultsdemonstratethatparticipantsfeltthatthetrainingwashighqualityandtheyfeltconfidentthattheycouldtraintheirstaffuponreturntotheirrespectivedistrictswiththeknowledgeandresourcesgained.Thisyear’sQTtrainingcycleincludedanoptionalafternoonsessionforanyinterestededucatorsonhowtoessentializegradelevelcontentstandardsandhowtodevelopcurriculumandprovideinstructionthatisalignedtothosestandardsforstudentswhoarefunctioningoffgradelevel,withafocusonstudentswithsignificantcognitivedisabilities(SWSCD).Weaskedparticipantstoratetheirconfidenceinusingtheknowledgeacquiredduringthesessionaswellastoevaluatethequalityofthepresentationandmaterials.Afour-pointscalewasemployed(StronglyDisagree,Disagree,Agree,StronglyAgree).ThesurveyfortheafternoonsessionwasconductedonlinewithQualtricssoftware.Percentagesofresponsesforeachstatementusedinthesurveyareprovidedbelow.Thefirsttableprovidesasummaryofthedatarelatedtoparticipantconfidence,whilethesecondprovidestheirevaluationofthequalityofthepresentation.Therespondentn-sizesrangedfrom26-30,dependinguponthequestion.Abargraphofstudyresultsisprovidedbelow,followedbytablesofdescriptivestatistics.Thedatavisualizationbelowwasconductedwithggplotinthetidyversepackage(Wickham,H.,2017).


Note:Thefirsttwographspresentparticipants’confidenceintheirabilitytousetheinformationpresentedintheareasofessentializationandcurriculumandassessment,respectively.Thenextfourgraphsconveyevaluationsofthepresentersforthecurricularandinstructionandessentializationtrainings.Resultsareverypositive,withsomereviewersfeelinglessconfidentabouttheirabilitiestotrainothersabouttheessentializationprocess.Thisoutcomewasexpected.Theprocessiscomplex,particularlygiventheunderstandingthatthiswasthefirsttimetheyhadreceivedsuchtraining.

Confidence_CI Confidence_Ess Eval_CI_SWSCD Eval_Ess Eval_GP_CI_SWSCD Eval_GP_Ess

0

20

40

60

Ratings by Group

Ave

rage

Gro

up R

atin

gs

ratingStrongly Agree

Agree

Disagree

Strongly Disagree


ConfidenceScalePercentagesStatementFollowingthecurriculumandinstructiontraining,Iamconfident:

StronglyDisagree Disagree Agree Strongly

Agree

1. Inmyunderstandingoftheessentializationofgradelevelacademiccontentstandards.

0 0 60 40

2. InmyabilitytoessentializetargetedELA,Math,orSciencecontentstandards.

0 10 56 33

3. Inmyabilitytoessentializecontentstandardssuchthattheyarealignedtoastudent’scurrentlevel(s)ofperformance.

0 6 63 30

4. InmyabilitytoaccessandusetheCurricularandInstructionalMaterialsforStudentswithSignificantCognitiveDisabilitiesonlinetrainingcourseandresources.

0 6 73 20

5. Inmyabilitytotrainothersintheessentializationofgradelevelcontentstandardsprocess.

0 31 51 17

6. InmyunderstandingoftheavailableC&Iresourcesandhowtoaccessthem.

0 3 65 30

7. InmyabilitytodevelopC&IlessonsinELAand/orMaththatarealignedtoEsSt.

0 3 76 19


StatementFollowingthecurriculumandinstructiontraining,Iamconfident:


Agree

8. InmyabilitytodevelopC&Ilessonsatmultiplelevelsofcomplexity.

0 3 76 19

9. ThatIcouldexplaintopeersandcolleagueshowtoaccessandusetheavailableC&IresourcesforSWSCD.

0 3 76 19

10. InmyabilitytoessentializestandardsanddevelopPLAAFP,IEPGoalsandObjectives,andC&IlessonsthatarealignedtothetargetedEsSt.

0 7 76 15

Note.ELA=Englishlanguagearts;C&I=CurriculumandInstruction;SWSCD=StudentswithSignificantCognitiveDisabilities;PLAAFP=PresentLevelsofAcademicAchievementandFunctionalPerformance;IEP=IndividualizedEducationProgram.Evaluation–EssentializationProcessandIntendedUsesPercentagesStatement


Agree1. Theinformationforthissectionofthetrainingwaswellorganizedandeasytofollow.

0 3 57 38

2. Thepresenter(s)wereknowledgeable,organized,andclear.

0 0 50 50

3. Thepresenterengagedparticipantstodiscussandapplywhatwewerelearning.

0 0 57 42

4. ThetrainingprovidedandreferencedresourcesthatIcanuseafterthesession.

0 0 42 57


Statement


Agree5. Ifeltcomfortablewiththeinformationandresourcespresented.

0 3 50 46

GuidedPracticeEssentializingStandards6. Theinformationforthissectionofthetrainingwaswellorganizedandeasytofollow.

0 3 53 42


0 3 53 42


0 0 61 38


0 0 50 50

10. Ifeltcomfortablewiththeinformationandresourcespresented.

0 3 57 38

Note.ELA=Englishlanguagearts;C&I=CurriculumandInstruction;SWSCD=StudentswithSignificantCognitiveDisabilities;PLAAFP=PresentLevelsofAcademicAchievementandFunctionalPerformance;IEP=IndividualizedEducationProgramEvaluation–CurricularandInstructionalResourcesforSWSCDPercentagesStatement


Agree1. Theinformationforthissectionofthetrainingwaswellorganizedandeasytofollow.

0 0 53 46


0 0 50 50


0 0 53 46


Statement


Agree4. ThetrainingprovidedandreferencedresourcesthatIcanuseafterthesession.

0 0 53 46


0 0 53 46

GuidedPracticeEssentializingStandards6. Theinformationforthissectionofthetrainingwaswellorganizedandeasytofollow.

0 0 50 50


0 0 50 50


0 0 57 42


0 0 50 50


0 0 57 42

Note.ELA=Englishlanguagearts;C&I=CurriculumandInstruction;SWSCD=StudentswithSignificantCognitiveDisabilities;PLAAFP=PresentLevelsofAcademicAchievementandFunctionalPerformance;IEP=IndividualizedEducationProgramInaddition,alltechnicalassistancequestionsthatwereceivefromthefieldaspartofourHelpDeskaredocumented.Thelogofthetechnicalassistanceprovisionisreviewedeachmonth,aswellasannually,inordertodeterminewhataspectsofourassessmentsystemneedfurtherclarificationorimprovement.TheHelpDesklogispublishedinAppendix2.3B.9.Oregonmonitorsthequalityofitssysteminseveralwaysinordertosupportcontinuousimprovement.Intermsoftheassessmentquality,itemstatisticsarereviewedeachyearanditemsthatarenotfunctioningasintendedareremovedandreplacedbybetterfunctioningfield-testitems.In2014-15,itemswerereviewedintwophases,firstusingclassicaltesttheory(CTT)and


secondusingRaschanalyses.Allitemsflaggedasaresultofthestatisticalreviewswereanalyzed,item-by-item,byateamofmeasurementandcontentexpertsatBRT.Notallflaggeditemswereremoved,asseveraldidnothaveapparentdesignflaws.Considerationsregardingdomainrepresentationaswellasitemdifficultyrangealsowereconsideredduringthereviewprocess.Wealsoemployeddifferentdecisionrulesforuniqueitemsversushorizontally-orvertically-scaledanchoritems.Itwasimportantinmanycasestomaintainanchoritems.Itemswithcleardesignflawswereremovedfromsubsequentanalysesandreporting.Thefollowingflaggingcriteriawereemployed:

CTT:Auniqueitemwasflaggedifithadap-valueof.10orlower,.90orhigher,orapointbiserial<.15.Anchoritemswereflaggediftheyhadap-valueof.10orloweror.95andhigheronallformsorapointbiserial<.45onanyform.Rasch:Uniqueitemswereflaggediftheiroutfitmeansquarevalueswerebetween0and.25or>1.5.Anchoritemswereflaggediftheiroutfitmeansquarevalueswere<.5,>1.8forhorizontalitems,or>2.0forverticalanchoritems.

Outofatotalof5,929itemsdevelopedin2014-15,166wereremoved(2.8%).WealsoimplementaconsequentialvaliditystudyeachyearthatsurveysQAsandQTsregardingtheacademicandsocialconsequencesoftheORExt,bothintendedandunintended.TheConsequentialValidityreportispublishedinAppendix2.3B.10.ODEandBRTstaffreviewtheresultsofthesurveyannuallytodeterminewhatprogramimprovementsareneeded.Asummaryoftheresultsisprovidedbelow.ODEimplementedaresearchsurveyprogramtoaddresstheneedtodocumenttheconsequences,bothintendedandunintended,oftheORExtAssessments.Theresearchquestionshavebeenframedbaseduponcurrentconsequentialvalidityapproachesforalternateassessmentsintheliterature,aswellasissuesthatareofspecificvalueinOregon.Thesurveyincluded344respondents.Thiswas25%ofthesolicitedrespondents,whowereallQualifiedAssessors(QAs)andQualifiedTrainers(QTs)intheor.k12test.comdatabase.Thesamplewas84%femaleandrepresentedallregionsofthestate,aswellasageranges.Thesurveyincludedarangeofquantitativeandqualitativecomponents.ThequantitativeresultsdemonstratethatQAsandQTscontinuetofeelthattheORExttestitemswereeasytoadministerandscore(58%StronglyAgree)andfeltconfidentintheirabilitytointerpretscaledscoresandAchievementLevelDescriptorsfortheORExt(54%Agree).Theyalsofeltthattheitemswereaccessibleforstudentswhoparticipated(53%Agree)andthattheORExtreflectedtheacademiccontentthatSWSCDshouldbelearning(57%Agree).QAsandQTsfeltmarginallypositiveabouttheeducationalimpactsoftheORExtandmarginallynegativeaboutitssocialimpacts.TheresultsagaindemonstratethattheORExtcontentareaassessmentsgenerallyrequireuptoonehourtoadminister.ThequalitativeresultsrevealedtwoareasinwhicheducatorsappreciatedtheORExtandfourareasofneededimprovement.QAsandQTssaidthattheyappreciated:1)theassessment'sefficiency(i.e.,morestreamlinedadministration,easeofadministration,easiertogiveandscoreonline,onlinematerialsdistribution);and,2)overallitemandtestdesign(i.e.,oneitemperpage,visualsupports,scoringprotocolandstudentmaterials


design,accessibilityoftestquestions).Teachersrecommendedthefollowingareasofimprovement,notallofwhichareactionable:1)Optiontoadministertheassessmentelectronically,2)Afunctionalskillsassessment,3)Newitemsforverylowfunctioningstudentsshouldbedeveloped,and4)Amathassessmentcomposedofmorepractical/lifeskillsproblemsinvolvingtimeandmoney.Completeresults,includinganticipatedresponses,fromthesurveycanbefoundinAppendix2.3B.10.

2.3CTechnology-basedAssessmentsTheORExtwasimplementedonasmallscaleusingatechnology-basedplatformaspartofPhase2oftheORExtPilotTabletAdministrationstudyconductedinthespring2017.Thecurrentplanistomakethetablet-basedadministrationoftheORExtavailablestatewidenextyear,2017-18.Acompletereportoftheresultsofthestudy,includingtheresearchplanandthelessonslearned,isprovidedinAppendix2.3C.Inshort,thisyear’stabletadministrationstudydemonstratedthatQAssupportatabletadministrationoftheORExtatthestatewidelevel.Thestudyresultsalsodemonstratedthatadditionaltrainingmustbeprovidedforthemanualwritingscoringprocess.Inaddition,theadministrationoftheORoraforstudentswhoseORExttestingisdiscontinuedaftertheyhavemettheminimumparticipationrulewillbeincorporatedintothetabletadministrationnextyear.Tosupporttrainingandunderstandingofthesystembybothteachersandstudents,practiceitemsinatabletformatwillbeprovidedforstakeholderstoprepareforfuturetablet-basedadministrations.Thisyear’sstudyaddressedissuesrelatedtodatabasecommunicationsystemstoensuredatasecurityandaccuratedatastorageandaccessatthedistrictlevel.Thepaper/pencilversionwillcontinuetobeavailableforstudentswhocannotaccessatabletadministration.2.4MonitoringTestAdministrationTheODEmaintainsarigoroustrainingsystemtosupportstandardizedtestadministrationfortheORExt,locatedathttps://or.k12test.com(securewebsite,butseescreenshotbelowforanexampleoftrainingcontent).


Theor.k12test.comwebsiteincludesatrainingsectionthataddressesanysystemsupdates,theprocessforbecomingaQualifiedAssessororQualifiedTrainer,studenteligibilityexpectations,studentconfidentialityandtestsecurity,testadministrationandscoringexpectations,examplesofappropriateandinappropriateadministration(video),supportingstudentaccesstoitemswithoutviolatingthetestconstruct,contentareatrainingsthatdemonstratehowtoadministeritemsinELA,Math,andScience(video,withsupportingtestmaterials),andhowtoaccesssecuretestsandcompletedataentry.InformationforQAs,QTs,andparentsregardingtheORExtisalsoprovided,asareallnecessarysupportmaterials.ForQAs,thesematerialsincludepracticeteststoprepareboththemselvesandstudentsfortheannualassessmentandallofthetrainingmaterialsusedonthewebsite.Inadditiontothesematerials,QTshaveaccesstoalltrainingmaterialsnecessarytoprovideannualtrainingtoQAsintheirpurview(seescreenshotbelow):


Inaddition,monitoringandreportingrelatedtotestadministrationissuesfortheORExtisaddressedviageneralODEreportingsystems.InformationregardingthisprocesscanbelocatedinthegeneralassessmentsystemPeerReviewevidencesubmission.2.5TestSecurity

2.5APreventionofAssessmentIrregularitiesTestsecuritypoliciesandconsequencesforviolationareaddressedintheTestAdministrationManualonanannualbasis(seeAppendix1.4.2,p.29-33).Thesepoliciesincludetestmaterialsecurity,propertestpreparationguidelinesandadministrationprocedures,consequencesforconfirmedviolationsoftestsecurity,andannualtrainingrequirementsatthedistrictandschoollevelsforallindividualsinvolvedintestadministration.Consequencesforadult-initiatedtestirregularitiesmaybesevere,includingplacingteachinglicensesinjeopardy(seeAppendix1.4.2,p.31-33).


2.5BDetectionofTestIrregularitiesTheODEutilizesalocalizedmonitoringsystemwhereschooltestcoordinatorsoverseebuilding-leveladministrationbytrained,QualifiedAssessors,andreporttocentralizeddistricttestcoordinators,whoarethenresponsibleforreportinganyconfirmedviolationstoODE.Improprietiesaredefinedasadult-initiatedorstudent-initiatedandinvestigatedaccordingly(seeAppendix1.4.2,p.29-31).

2.5CRemediationFollowingTestSecurityIncidentsODE'salternateassessmentprogrammanagerinvestigatesandremediatessubstantiatedtestsecurityincidentsfortheORExtbyworkingwithdistricttestcoordinators.AdditionalinformationregardingthisprocesscanbelocatedinthegeneralassessmentsystemPeerReviewevidencesubmission.

2.5DInvestigationofTestIrregularitiesSchoolanddistricttestcoordinatorsconductinitialinvestigationsintoallallegedtestirregularities.OncereportedtoODE,allallegedtestirregularitiesareinvestigatedinconsultationwithdistricttestcoordinatorsandthetestvendor,asappropriate(seeAppendix1.4.2,p.31-33).Intheeventthatatestirregularityisdeterminedtobefactual,consequencesaredeterminedbaseduponcontextualissuesthatarebroughttolightduringtheinvestigation.AdditionalinformationregardingthisprocesscanbelocatedinthegeneralassessmentsystemPeerReviewevidencesubmission.2.6SystemsforProtectingDataIntegrityandPrivacy

2.6AIntegrityofTestMaterialsTestmaterialsfortheORExtaremaintainedthroughoutdevelopment,dissemination,andadministrationviamultiplemechanisms.AllitemsunderdevelopmentarestoredinsecurefileserversmanagedbyBehavioralResearch&TeachingattheUniversityofOregon,thetestvendorfortheORExt.Itemreviewsnecessarytoprovidealignment,bias,andsensitivityinformationareconductedonlineusingthesecureDistributedItemReview(DIR)platformhttp://brtitemreview.com(securewebsite,butseeAppendix3.1Bforasystemoverview).Forthe2016-2017schoolyear,allsecuretestdistributionanddataentrywashostedbyODE'ssecurefiletransfersystem,whichisapassword-protectedtestdistributionanddataentrysystemlocatedathttps://district.ode.state.or.us/apps/login/(securewebsite,butseescreenshotbelowforreference).AdataentryguideisprovidedinAppendix2.6.


AdditionalinformationregardingtestsecuritycanbelocatedinthegeneralassessmentsystemPeerReviewevidencesubmission.

2.6BSecureStudent-LevelAssessmentDataStudentleveldataisprotectedbyrelevanttrainingandthroughasecuredatasysteminwhichalldataentryisconductedonlineusingpassword-protected,secureproceduresonthehttps://or.k12test.comorhttps://district.ode.state.or.us/apps/login/websites,asidentifiedabove.Onlytraineduserswithavestededucationalinterestwhohavesignedtestsecurityagreementsareauthorizedtoaccesstoonlinedataentrysystems.SeeAppendix2.6foradditionaldataentryexpectationsfor2016-17.


2.6CProtectingPersonallyIdentifiableInformationAllconfidential,personallyidentifiablestudentinformationisprotectedbypolicyandsupportedbytraining(seeAppendix1.4.2,p.26).Theminimumnumberofstudentsnecessarytoallowreportingofstudentsandstudentsubgroupsvariesbyrating(i.e.,achievement,growth,graduation,andschoolsize)bylevel(i.e.,school/district/state),andbynumberofyearsofassessmentdataavailable.Forexample,toreceiveanachievementrating,schoolsmusthaveatleast40testsforthetwomostrecentschoolyearsinreadingormathematics.Alternatively,smallschoolsreceiveanachievementratingiftheyhaveatleast40testsoverthemostrecentfouryears.Ifaschooldoesnothaveatleast40testsoverafour-yearperiod,theywillnotreceiveanachievementscore(seeAppendix2.6C).Similarrulesareappliedtostudentsubgroups,includingstudentswithdisabilities,Englishlearners,andstudentsfromdiverseracial/ethnicbackgrounds(seeAppendix2.6C,p.7).

2016-2017 OR Extended Assessment Critical Element 3 – Page

46

CriticalElement3–TechnicalQuality:Validity3.1OverallValidity,IncludingValidityBasedonContentAselaboratedbyMessick(1989)1,thevalidityargumentinvolvesaclaimwithevidenceevaluatedtomakeajudgment.Threeessentialcomponentsofassessmentsystemsarenecessary:(a)constructs(whattomeasure),(b)theassessmentinstrumentsandprocesses(approachestomeasurement),and(c)useofthetestresults(forspecificpopulations).Validationisajudgmentcallonthedegreetowhicheachofthesecomponentsisclearlydefinedandadequatelyimplemented.Validityisaunitaryconceptwithmultifacetedprocessesofreasoningaboutadesiredinterpretationoftestscoresandsubsequentusesofthesetestscores.Inthisprocess,wewantanswersfortwoimportantquestions.Regardlessofwhetherthestudentstestedhavedisabilities,thequestionsareidentical:(1)Howvalidisourinterpretationofastudent'stestscore?and(2)Howvalidisittousethesescoresinanaccountabilitysystem?Validityevidencemaybedocumentedatboththeitemandtotaltestlevels.WeusetheStandards2(AERAetal.,2014)indocumentingevidenceoncontentcoverage,responseprocesses,internalstructure,andrelationstoothervariables.Thisdocumentfollowstheessentialdatarequirementsofthefederalgovernmentasneededinthepeerreviewprocess.3ThecriticalelementshighlightedinSection4inthatdocument(withexamplesofacceptableevidence)include(a)academiccontentstandards,(b)academicachievementstandards,(c)astatewideassessmentsystem,(d)reliability,(e)validity,and(f)otherdimensionsoftechnicalquality.Inthistechnicalreport,dataarepresentedtosupporttheclaimthatOregon’sAA-AAASprovidesthestatetechnicallyadequatestudentperformancedatatoascertainproficiencyongradelevelstatecontentstandardsforstudentswithsignificantcognitivedisabilities–whichisitsdefinedpurpose.TheAA-AAASarelinkedtogradelevelacademiccontent,generatereliableoutcomesatthetestlevel,includeallstudents,haveacogentinternalstructure,andfitwithinanetworkofrelationswithinandacrossvariousdimensionsofcontentrelatedtoandrelevantformakingproficiencydecisions.SampleitemsthatconveythedesignandsamplecontentofORExtitemsareprovidedinAppendix2.2.3.Theassessmentsareadministeredandscoredinastandardizedmanner.AssessorswhoadministertheORExtaretrainedtoprovidethenecessarylevelofsupportforappropriatetestadministrationonanitem-by-itembasis.Therearefourlevelsofsupportoutlinedintraining:fullphysicalsupport,partialphysicalsupport,promptedsupport,andnosupport.Itemsweredesignedtodocumentstudents’skillandknowledgeongradelevelacademiccontentstandards,withthelevelofsupportprovideddesignednottointerferewiththe

1Messick,S.(1989).Validity.InR.L.Linn(Ed.),Educationalmeasurement(3rded.,pp.13-103).NewYork:

AmericanCouncilonEducation.2AmericanEducationalResearchAssociation(AERA),AmericanPsychologicalAssociation,&National

CouncilonMeasurementinEducation(2014).Standardsforeducationalandpsychologicaltesting.Washington,DC:AERA.

3U.S.DepartmentofEducation(2015).PeerReviewofStateAssessmentSystems:Non-RegulatoryGuidanceforStatesforMeetingtheRequirementsoftheElementaryandSecondaryEducationActof1965,asAmended.


47

constructbeingmeasured.OnlyonetestadministrationtypeisusedfortheORExt,patternedaftertheformerScaffoldversionoftheassessment.Assessorsadministerthepromptandifthestudentdoesnotrespond,theAssessorreadsadirectivestatementdesignedtofocusthestudent'sattentionuponthetestitemandthenrepeatstheprompt.Ifthestudentstilldoesnotrespond,theAssessorrepeatsthepromptasneededandotherwisescorestheitemasincorrectandmovesontothenextitem.TrainingdocumentationisprovidedinAppendices2.3B.1-2.3B.8.Giventhecontent-relatedevidencethatwepresentrelatedtotestdevelopment,alignment,training,administration,scoring,thereliabilityinformationreflectedbyadequatecoefficientsfortests,and,finally,therelationoftestsacrosssubjectareas(providingcriterion-relatedevidence),weconcludethatthealternateassessmentjudgedagainstalternateachievementstandardsallowsvalidinferencestobemadeonstateaccountabilityproficiencystandards.

3.1AAlignmentBetweenAA-AAASandAcademicContentStandards

OurfoundationofvalidityevidencefromcontentcoveragefortheORExtcomesintheformoftestspecifications(seeAppendix2.1)andtestblueprints(seeAppendix2.1B).Amongotherthings,theStandards(AERAetal.,2014)2suggestspecificationsshould“definethecontentofthetest,theproposedtestlength,theitemformats…”(Standard4.2,p.85).2Allitemsarelinkedtogradelevelstandardsandaprototypewasdevelopedusingprinciplesofuniversaldesign4withtraditional,content-referencedmultiple-choiceitemwritingtechniques5.Themostimportantcomponentintheseinitialstepsaddressedlanguagecomplexityandaccesstostudentsusingbothreceptive,aswellasexpressive,communication.Additionally,bothcontentbreadthanddepthwereaddressed.WedevelopedonetestformfortheORExtthatutilizesascaffoldapproach.Thisapproachallowsforstudentswithverylimitedattentiontoaccesstestcontent,whilethesupportsarenotutilizedforstudentswhodonotneedthissupport.Wedevelopedthetestiterativelybydevelopingitems(seeAppendix2.2.1,whichconveysouritemwritertrainingmaterials),pilotingthem,reviewingthem,andeditingsuccessivedrafts.WeusedacombinationofexistingpanelsofveteranteacherswhohaveworkedwiththeOregonDepartmentofEducation(ODE)invariousadvisingrolesontestingcontentingeneralandspecialeducation,usingthesameprocessesandcriteria,aswellastheintroductionofnewerteacherswhoarequalifiedasweproceedtoremainrelevant.BehavioralResearchandTeaching(BRT)personnelconductedtheinternalreviewsofcontent.Aftertheinternaldevelopmentofprototypeitems,allreviewstheninvolvedOregoncontentandspecialeducationexpertswithsignificanttrainingandK-12classroomexperience.

4Johnstone,C.,Thompson,S.,Bottsford-Miller,N.,&Thurlow,M.(2008).Universaldesignandmultimethodapproachestoitemreview.EducationalMeasurement,27(1),26-36.5Halydyna,T.,&Rodriquez,M.C.(2013).Developingandvalidatingtestitems.NewYork,NY:Routledge.


48

TheORExtincorporatescontinuousimprovementintoitstestdesignviafield-testinginallcontentareasonanannualbasis,withanaverageof25%newitems.Theseitemsarecomparedtooperationalitemsbasedonitemfunctioningandtestdesignfactors,generatingdatausedtoreplaceitemsonanannualbasis,incorporatingthenewitemsthatfillaneededgapwithregardtocategoricalconcurrence,orprovideforawiderrangeoffunctioningwithregardtocomplexitylevels:low–medium–high,comparabletoWebb’sDOK(seeSection3.1A).BRTemployedamulti-stagedevelopmentprocessin2014-15toensurethattestitemswerelinkedtorelevantcontentstandards,wereaccessibleforstudentswithsignificantcognitivedisabilities,andthatanyperceiveditembiaseswereeliminated.Theitemreviewprocessincluded51reviewerswithanaverageof22yearsofexperienceineducation.TheORExtassessmentshavebeendeterminedtodemonstratestronglinkagetogradelevelacademiccontent,overall.Fulldocumentationoftheinitial2014linkagestudyandanew,independentalignmentstudyconductedinspring,2017isprovidedinAppendix3.1A.Noitemdevelopmentwasrequiredin2016-17.Thesummarysectionoftheindependentalignmentstudyreportstatesthat,“Oregon’sExtendedAssessments(ORExt)inEnglishLanguageArts,Mathematics,andSciencewereevaluatedinalow-complexityalignmentstudyconductedinSpringof2017.Averagesofreviewerprofessionaljudgmentsoverfiveseparateevaluationsweregathered,reviewed,andinterpretedinthepagesthatfollow.Inthethreeevaluationsthatinvolveddeterminingtherelationshipbetweenstandardsanditems,reviewersidentifiedsufficienttostrongrelationshipsamongassessmentcomponentsinallgradesandallsubjectareas.InthetwoevaluationsinvolvingAchievementLevelDescriptors,reviewersidentifiedthirtyinstancesofsufficienttostrongrelationshipsoutofthirty-fourpossiblerelationshipopportunitiesresultinginanoverallaffirmedrelationshipwithareasforrefinementsidentified.”BecausetheassessmentsdemonstratesufficienttostronglinkagetoOregon'sgeneraleducationcontentstandardsanddescriptivestatisticsdemonstratethateachcontentareaassessmentisfunctioningasintended,itisappropriatetodeducethatthesestandardsdefinetheexpectationsthatarebeingmeasuredbytheOregonExtendedassessments.TheOregonExtendedassessmentsyieldscoresthatreflectthefullrangeofachievementimpliedbyOregon'salternateachievementstandards.EvidenceofthisclaimisfoundinthestandardsettingdocumentationsubmittedinSection6.2.StandardsweresetforallsubjectareasonJune15-17,2015.Standardsincludedachievementleveldescriptorsandcutscores,whichdefineOregon'snewalternateachievementstandards(AAS).TheStateBoardofEducationofficiallyadoptedtheAASonJune25,2015.

3.1BAA-AAASLinkagetoGeneralContentStandardsCompleteresultsoftheanalysisofthelinkageofthenewEssentializedAssessmentFrameworks,(EAF),composedofEssentializedStandards(EsSt),togradelevelCCSSinEnglishlanguageartsandmathematicsandlinkedtoORSciandNGSSinscience,arepresentedinSection3.1A.TheclaimisthattheEsStaresufficientlylinkedtogradelevelstandards,whiletheORExtitemsarealignedtotheEsSt.Inadditiontopresentinglinkage


49

informationbetweengradelevelcontentstandardsandtheEsSt,thelinkagestudypresentsalignmentinformationrelatedtotheitemsonthenewORExtincomparisontotheEsSt.Extendedassessmentshavebeendeterminedtolinksufficientlytogradelevelacademiccontentstandards.Fieldtestitemsareaddedeachyearbasedonitemalignmenttostandards.TheOregonExtendedassessmentslinktogradelevelacademiccontent,asreflectedintheitemdevelopmentprocess.OregonalsohadeachoperationalitemusedontheOregonExtendedassessmentevaluatedforalignmentaspartoftwocomprehensivelinkagestudies,oneperformedin2014andanindependentalignmentstudyperformedin2017(seeSection3.1A).Theprofessionalreviewersinaninternalstudyin2014andanindependentstudyinspring2017includedbothspecialandgeneraleducationexperts,withcontentknowledgeandexperienceinadditiontospecialeducationexpertise.Accordingtotheindependentlinkagestudyreport(seeAppendix3.1A),thespring2017reviewwasconductedbyexpertreviewerswithprofessionalbackgroundsineitherSpecialEducation(thepopulation),Assessment,orinOregon’sadoptedcontentstandards.Reviewerswereassignedtoreviewgrade-levelitemsrelativetotheirexperienceandexpertise.Inall,39reviewersparticipated.Thirty-four(34)participatedinall5evaluations:thirteen(13),fortheEnglishLanguageArtsreview,fifteen(15)fortheMathematicsreview,andsix(6)fortheSciencereview.AllparticipantswereassignedtoatleastonespecificcontentareaasshowninTable1.Note:Fourindividualswereassignedtotwoareasofreview.Thethirty-nineindividualswhoparticipatedinthestudyhadarobustlegacyofexperienceinthefieldandinthestate.Participantsrepresented25uniqueschooldistrictsacrossthestaterepresentingbothurbanandruralperspectives.All39oftheindividualsparticipatinginthestudyheldcurrentteachinglicenses.Twoindividualsalsoheldadministrativelicenses.Yearsofexperienceintheirarearangedfrom3–30yearsofexperiencewithanaverageof17yearsofexperience.(Mode=11years,Median=16years).Oneindividualindicated50yearsofexperienceinthefield.Threeofthe39individualsheldaBachelor’sdegreeonly.Thirty-sixheldaBachelor’sdegreeandatleastoneMaster’sdegree.TwoheldaBachelor’sdegree,atleastoneMaster’sdegree,andadoctoraldegree.Fourteen(36%)oftheindividualsidentifiedasexpertsinaspecificContentareaand25(64%)oftheindividualsidentifiedSpecialeducationastheirprimaryareaofexpertise.Theseskilledreviewersweretrainedbysynchronouswebinarsonlinkage/alignment,aswellasitemdepth,breadth,andcomplexityandthencompletedtheirratingsonlineviaBRT'sDistributedItemReview(DIR)websiteandonExcelspreadsheetssharedwiththeresearcherelectronically(seeAppendix3.1Bforanoverview).Mocklinkageratingswereconductedinordertoaddressquestionsandensureappropriatecalibration.Reviewersratedeachessentializedstandardona3-pointscale(0=nolink,1=sufficientlink,2=stronglink)asitrelatedtothestandardthetestdevelopershaddefinedforthatessentializedstandard.Itemswereevaluated,inturn,basedupontheiralignmenttotheessentializedstandardona3-pointscale(0=insufficientalignment,1=sufficientalignment,2=strongalignment).Whenaveragedacrossreviewers,1.00-1.29was


50

consideredinthelowrange,1.30–1.69wassufficient,and1.70–2.0wasstrong.Additionalcommentwasrequestedforanyessentializedstandardoritemwhoselinkagewasrated0.Overall,the2017independentalignmentstudyconcludesthat:“First,reviewerswereaskedtoconductanaffirmationalreviewoftherationaleusedbytestdeveloperstoomitcertaincontentstandards.ThisfindingwasusedtoinferthatthefinalstandardsselectedforinclusionoromissioninOregon’sExtendedAssessmentwerechosenrationallyandthatthefinalscopeofcontentstandardscanbeconsideredjustifiableforthepopulationforthesubjectarea. Conclusion:Thisreview,withalowestaveragerateof.82(onascaleof1),permits theinference:thescopeofthestandardsselectedfortranslationtoEssentialized Standardswererationallyselected.Noneofthestandardsde-selected(for inaccessibilityorforbeingcoveredelsewhere)werestronglyidentifiedforre- inclusion,norwereidentifiedasacriticalholeforthispopulationofstudents.Second,reviewerswereaskedtoidentifythestrengthofthelinkbetweenthesourcestandardandtheEssentializedStandard.ThisfindingwasusedtoinferthattheprocessundertakentoessentializeagivenSourceStandarddidnotfundamentallyorcriticallyaltertheknowledgeorskillsetintendedbythesourcestandardforthispopulationofstudents(furtherconfirmingthatthecontentselectedforassessmentiscomparable). Conclusion:Thisreview,witharangeof1.5–1.9(onascaleof2)permitsthe inference:theEssentializedStandardswerefoundtolinksufficientlytothesource standardsonaveragebeyondthe“sufficient”averageof1.0.Third,reviewerswereaskedtoidentifythestrengthofthealignmentbetweentheEssentializedStandardsandtheitemsandtoreviewtheitemsdevelopedusingtheEssentializedStandardsforbias,andaccessibility.Thefindingfromthisreviewwasusedtoinferthattheitemswrittenforthisgradeandsubjectarea(usingtheseEssentializedStandards)wereadequatelylinkedtotheEssentializedStandards,werefreefrombias,andwereaccessibletostudentswithsignificantcognitivedisabilities. Conclusion:Thealignmentreview(1.32–1.89),accessibilityreview(.67**–1.0), andfreedomfrombiasreview(.65**–1.0)allpermittheinferencethatthetest itemsindicatearelationshipwiththesourcestandards,thetestitemsarenotoverly biasedtowardsoragainstanyparticulargroupofindividuals,andthetestitemsare writtensuchthatthecontentandintentcanbeaccessedbystudentswiththemost significantcognitivedisabilities.(**Note:thisrangewasskewedbyfeedbackfrom onereviewer--ELA-Grade3–whosecommentswerenotedinthisstudy.Removing thatindividual’scommentswouldresultinarangeof.90–1.0accessibilityrange and.89–1.0freedomfrombiasrangerespectively.)Fourth,reviewerswereaskedtoreviewthestatementsusedtodescribestudentachievementonthetest(theAchievementLevelDescriptors)andtheiralignmenttotheEssentializedStandardsthatthestudentsweretestedon.ThefindingfromthisreviewwasusedtoinferthattheskillsandachievementsdescribedbytheAchievementLevelDescriptorsforeachsubjectandgradelevelarealignedwiththecontentstandardbeingmeasured. Conclusion:Thereviewsrangingfrom.68*–1.0permittheinferencethatthe descriptionsmaderegardingstudentskillsetareanaccuratereflectionofthe


51

standardsfromwhichtheassessmentwasdevelopedatallthreelevelsevaluated. (*OneoutlierforELA-Grade4providedareviewofa.52average).Fifth,andfinally,reviewerswereaskedtoreviewthealignmentoftheAchievementLevelDescriptorstotheitems.Thefindingfromthisreviewwasusedtoinferthateachiteminthedevelopedassessment(s)wasappropriatelyalignedtoitsassociatedAchievementLevelDescriptor(furtherconfirmingthatdecisionsmadeusingthistestwerealignedwiththeintentofthesourcestandard). Conclusion:Fourteenoftheseventeengrade-levelreviewsresultedinanaverage reviewerrangeof.67–1.0indicatinganappropriatealignmentbetweenALDsand theitemsaswritten.Thisreviewpermitstheinferencethat,overall,the AchievementLevelDescriptorsareaccuratereflectionsoftheitems.Inthree instances(Mathematics-Grades3and4,andELA-Grade8)theaveragealignmentby reviewerwas.5(indicatingthatoneofthetwoindividualsinthatcategorydidnot agreethattheitemsandALDswerealigned).” 3.2ValidityBasedonCognitiveProcessesEvidenceofcontentcoverageisconcernedwithjudgmentsabout“theextenttowhichthecontentdomainofatestrepresentsthedomaindefinedinthetestspecifications”(AERAetal.,2014,Standard4.12,p.89)7.Asawhole,theORExtiscomprisedofsetsofitemsthatsamplestudentperformanceontheintendeddomains.Theexpectationisthattheitemscoverthefullrangeofintendeddomains,withasufficientnumberofitemssothatscorescrediblyrepresentstudentknowledgeandskillsinthoseareas.Withoutasufficientnumberofitems,thepotentialexistsforavaliditythreatduetoconstructunder-representation(Messick,1989)6.TheORExtassessmentisbuiltuponavarietyofitemsthataddressawiderangeofperformanceexpectationsrootedintheCCSS,NGSS,andORScicontentstandards.ThechallengebuiltintothetestdesignisbasedfirstuponthecontentwithineachstandardinEnglishlanguagearts,mathematics,andscience.ThatcontentisRDBCinamannerthatisverifiedbyOregongeneralandspecialeducationteacherstodevelopassessmenttargetsthatareappropriateforstudentswiththemostsignificantcognitivedisabilities.Ourassessmentsutilizeuniversaldesignprinciplesinordertoincludeallstudentsintheassessmentprocess,whileeffectivelychallengingthehigherperformingstudents.ForstudentswhohaveverylimitedtonocommunicationandareunabletoaccesseventhemostaccessibleitemsontheORExt,anOregonObservationalRatingAssessment(ORora)wasfirstimplementedin2015-16.TheORoraiscompletedbyteachersanddocumentsthestudent'slevelofcommunicationcomplexity(expressiveandreceptive),aswellaslevelofindependenceinthedomainsofattention/jointattentionandmathematics.AcompletereportofORoraresultsfrom2016-17isprovidedinAppendix5.1D.Fifty-onereviewersanalyzedallORExtitemsforbias,sensitivity,accessibilitytothestudentpopulation,andalignmenttotheEssentializedStandards.Atotalof21reviewerswereinvolvedintheEnglishlanguageartsitemreviews.Anadditional21reviewerswere6Messick,S.(1989).Validity.InR.L.Linn(Ed.),Educationalmeasurement(3rded.,pp.13-103).NewYork:

AmericanCouncilonEducation.


52

involvedintheMathematicsitemreviews.Scienceemployedninereviewers.Reviewerswereorganizedintogradelevelteamsoftwospecialeducatorsandonecontentspecialist.SubstantiveevidencethathasbeendocumentedsuggeststhattheORExtitemsaretappingtheintendedcognitiveprocessesandthattheitemsareattheappropriategradelevelthroughthelinkage/alignmentstudiesdocumentedabove,includingreviewsoflinkage,contentcoverage,anddepthofknowledge.AcomprehensivereportoftheitemreviewprocessisavailableinAppendix3.1A.3.3ValidityBasedonInternalStructure(ContentandFunction)TheOregonExtendedassessmentsreflectpatternsofemphasisthataresupportedbyOregoneducatorsasindicatedbythefollowingthreetablesthathighlightthebalanceofstandardrepresentationbygradelevelforEnglishlanguagearts,mathematics,andscienceontheORExt.Therepresentationratioscanbecalculatedbydividingthestandardsbythetotalwithineachrespectivecolumn.Forexample,inGrade3Reading,approximately25%oftheitemsareintheReadingStandardsforLiteraturedomain,asthatdomainhas4writtenEssentializedStandards(EsSt)outofthetotalof16(4/16=25%).ThetestblueprintsbelowdirectlycorrespondtothenumberofESwrittenineachdomainwithintheEssentializedAssessmentFrameworks(EAF)spreadsheets.ThereareadditionalgradelevelstandardsaddressedbytheEsSt,assomeEsStlinktomultiplegradelevelcontentstandards.However,theblueprintsbelowreflectonlythewrittenEsStandarethusanunderrepresentationofthebreadthofgradelevelcontentaddressedbytheORExt.EnglishLanguageArtsDomain Grade3 Grade4 Grade5 Grade6 Grade7 Grade8 Grade11RF 2 2 2 RI 4 4 4 5 5 5 5RL 4 4 4 5 5 5 5WR 4 4 4 4 4 4 4LA 2 2 2 2 2 2 2TOTAL 16 16 16 16 16 16 16Note.RF=ReadingStandards:FoundationalSkills.RI=ReadingStandardsforInformationalText.RL=ReadingStandardsforLiterature.WR=Writing.LA=Language.


53

MathematicsDomain Grade3 Grade4 Grade5 Grade6 Grade7 Grade8 Grade11OAT 7 4 3 NBT 2 6 8 NOF 3 8 6 MED 8 5 4 GEO 2 3 2 3 3 4 7RPR 3 2 TNS 9 7 2 EXE 6 2 6 STP 5 6 3 5FUN 4 7NAQ 2ALG 2TOTAL 22 26 23 26 20 19 23Note.OAT=OperationsandAlgebraicThinking.NBT=NumbersandOperationsinBaseTen.NOF=NumbersandOperations–Fractions.MED=MeasurementandData.GEO=Geometry.RPR=RatioandProportionalRelationships.TNS=TheNumberSystem.EXE=ExpressionsandEquations.STP=StatisticsandProbability.FUN=Functions.NAQ=NumbersandQuantities.ALG=Algebra.Science

Domain Grade5 Grade8 Grade11LFS 4 9 8PHS 4 7 9ESS 4 6 6ETS 2 2 TOTAL 14 24 23Note.LFS=LifeScienceStandards.PHS=PhysicalSciences.ESS=EarthandSpaceSciences.ETS=Engineering,Technology,andApplications.TheprimarypurposeoftheORExtassessmentistoyieldtechnicallyadequateperformancedataongradelevelstatecontentstandardsforstudentswithsignificantcognitivedisabilitiesinEnglishlanguagearts,mathematics,andscienceatthetestlevel.Allscoringandreportingstructuresmirrorthisdesignandhavebeenshowntobereliablemeasuresatthetestlevel(seeSection4.1).Theprocessofaddressinganygapsorweaknessesinthesystemisaccomplishedviafield-testing(seeSection3.1A).

PointMeasureCorrelationsDistributionsofpointmeasurecorrelationsandoutfitmeansquarestatisticsforoperationalitemsareprovidedbelow,bycontentareaandgrade.Pointmeasure


54

correlationsdisplayhowtheitemscorescorrelatewiththelatentoverallscore,whileoutfitmeansquarestatisticscloserto1.0denoteminimaldistortionofthemeasurementsystem.Allitemsincludedinthe2016-17operationalassessmentarerepresented.PointmeasurecorrelationsinELArangedfrom0.42to0.74.Alldatavisualizationswereconductedwithggplotinthetidyversepackage(Wickham,H.,2017).

English Language Arts − Grade 3

Point Measure Correlations

Item

Fre

quen

cies

0.4 0.5 0.6 0.7 0.8

02

46

8



Item

Fre

quen

cies

0.4 0.5 0.6 0.7 0.80

24

68



Item

Fre

quen

cies

0.4 0.5 0.6 0.7 0.8

02

46

8



Item

Fre

quen

cies

0.4 0.5 0.6 0.7 0.8

02

46

8



Item

Fre

quen

cies

0.4 0.5 0.6 0.7 0.8

02

46

8



Item

Fre

quen

cies

0.4 0.5 0.6 0.7 0.8

02

46

8


55



Item

Fre

quen

cies

0.2 0.3 0.4 0.5 0.6 0.7 0.8

02

46

8


56

Point-measurecorrelationsinmathematicsrangedfrom0.25to0.65.

Mathematics − Grade 3


Item

Fre

quen

cies

0.2 0.3 0.4 0.5 0.6 0.7 0.8

02

46

8



Item

Fre

quen

cies

0.2 0.3 0.4 0.5 0.6 0.7 0.8

02

46

8



Item

Fre

quen

cies

0.2 0.3 0.4 0.5 0.6 0.7 0.8

02

46

8



Item

Fre

quen

cies

0.2 0.3 0.4 0.5 0.6 0.7 0.8

02

46

8



Item

Fre

quen

cies

0.2 0.3 0.4 0.5 0.6 0.7 0.8

02

46

8



Item

Fre

quen

cies

0.2 0.3 0.4 0.5 0.6 0.7 0.8

02

46

8


57

Point-measurecorrelationsinsciencerangedfrom0.47to0.72.



Item

Fre

quen

cies

0.2 0.3 0.4 0.5 0.6 0.7 0.8

02

46

8

Science − Grade 5


Item

Fre

quen

cies

0.3 0.4 0.5 0.6 0.7 0.8

02

46

8


58

Science − Grade 8


Item

Fre

quen

cies

0.3 0.4 0.5 0.6 0.7 0.8

02

46

8

Science − Grade 11


Item

Fre

quen

cies

0.3 0.4 0.5 0.6 0.7 0.8

02

46

8


59

OutfitMeanSquareDistributionsOutfitmeansquarevaluesbelow1.0demonstratethatvaluesaretoopredictableandperhapsredundant,whilevaluesabove1.0indicateunpredictability.Itemsabove2.0aredeemedinsufficientformeasurementpurposesandflaggedforreplacement.WhilemostOMSvaluesinELAwerebetween0.5and1.5,oneiteminGrade6wasabove2.0andwasremoved.


Outfit Mean Squares

Item

Fre

quen

cies

0.0 0.5 1.0 1.5 2.0 2.5 3.0

01

23

45

6


Outfit Mean Squares

Item

Fre

quen

cies

0.0 0.5 1.0 1.5 2.0 2.5 3.0

01

23

45

6


Outfit Mean Squares

Item

Fre

quen

cies

0.0 0.5 1.0 1.5 2.0 2.5 3.0

01

23

45

6


Outfit Mean Squares

Item

Fre

quen

cies

0.0 0.5 1.0 1.5 2.0 2.5 3.0

01

23

45

6


Outfit Mean Squares

Item

Fre

quen

cies

0.0 0.5 1.0 1.5 2.0 2.5 3.0

01

23

45

6


Outfit Mean Squares

Item

Fre

quen

cies

0.0 0.5 1.0 1.5 2.0 2.5 3.0

01

23

45

6


60


Outfit Mean Squares

Item

Fre

quen

cies

0.0 0.5 1.0 1.5 2.0 2.5 3.0

01

23

45

6


61

WiththeexceptionofasingleiteminGrade7,whichwasreplaced,mathematicsOMSvaluesrangedfrom.35to1.7,demonstratingthattheitemsareperformingwithinexpectedranges.


Outfit Mean Squares

Item

Fre

quen

cies

0.0 0.5 1.0 1.5 2.0

01

23

45

6


Outfit Mean SquaresIte

m F

requ

enci

es

0.0 0.5 1.0 1.5 2.0

01

23

45

6Mathematics − Grade 5

Outfit Mean Squares

Item

Fre

quen

cies

0.0 0.5 1.0 1.5 2.0

01

23

45

6


Outfit Mean Squares

Item

Fre

quen

cies

0.0 0.5 1.0 1.5 2.0

01

23

45

6


Outfit Mean Squares

Item

Fre

quen

cies

0.0 0.5 1.0 1.5 2.0

01

23

45

6


Outfit Mean Squares

Item

Fre

quen

cies

0.0 0.5 1.0 1.5 2.0

01

23

45

6


62

Withtwoexceptions,oneiteminGrade5ScienceandanotherinGrade8,ScienceOMSvaluesrangedfrom.47to1.75,demonstratingthattheitemsareperformingwithinexpectedrangeswithafewexceptionsthatwillberemoved.


Outfit Mean Squares

Item

Fre

quen

cies

0.0 0.5 1.0 1.5 2.0 2.5 3.0

01

23

45

6

Science − Grade 5

Outfit Mean Squares

Item

Fre

quen

cies

0.0 0.5 1.0 1.5 2.0 2.5

02

46

8


63

Science − Grade 8

Outfit Mean Squares

Item

Fre

quen

cies

0.0 0.5 1.0 1.5 2.0 2.5

02

46

8

Science − Grade 11

Outfit Mean Squares

Item

Fre

quen

cies

0.0 0.5 1.0 1.5 2.0 2.5

02

46

8


64

AnnualMeasureableObjectivesFrequencies&PercentagesAnnualMeasurableObjective(AMO)calculationswereconductedbaseduponstudentperformanceontheORExttiedtotheverticalscaleusingRaschmodeling.Overallresultsarelargelyconsistentwith2015-16,withapproximately50%ofstudentswithsignificantcognitivedisabilitiesachievingproficiencyacrossgradesandcontentareas.ELAresultsarepresentedinblue,mathematicsindarkgreen,andscienceinred.Thedatavisualizationspresentedbelowwereconductedwithggplotinthetidyversepackage(Wickham,H.,2017).

15.16%

38.22%

29.82%

16.80%

0

50

100

150

200

1 2 3 4AMO Performance Level

Frequency

Grade 3 ELA

21.78%

24.36%

26.76% 27.10%

0

50

100

150


Frequency

Grade 4 ELA


65

25.04%

30.59%

23.70%

20.67%

0

50

100

150


Frequency

Grade 5 ELA

25.19% 25.00%25.95%

23.85%

0

50

100


Frequency

Grade 6 ELA


66

26.08%

17.65%

28.24% 28.04%

0

50

100

150


Frequency

Grade 7 ELA

32.02%

18.97%

22.92%

26.09%

0

50

100

150


Frequency

Grade 8 ELA


67

17.09%

24.10%

9.74%

49.06%

0

100

200

300


Frequency

Grade 11 ELA

30.28%

23.13%

38.27%

8.32%

0

50

100

150

200


Frequency

Grade 3 Math


68

29.64%

35.60%

26.06%

8.69%

0

50

100

150

200


Frequency

Grade 4 Math

26.30%

34.17%33.17%

6.37%

0

50

100

150

200


Frequency

Grade 5 Math


69

42.45%

12.05%

34.23%

11.28%

0

50

100

150

200


Frequency

Grade 6 Math

47.74%

3.54%

36.74%

11.98%

0

50

100

150

200

250


Frequency

Grade 7 Math


70

39.72%

18.97%

37.15%

4.15%

0

50

100

150

200


Frequency

Grade 8 Math

34.36%

21.54%

35.38%

8.72%

0

50

100

150

200


Frequency

Grade 11 Math


71

34.01%

15.15%

21.38%

29.46%

0

50

100

150

200


Frequency

Grade 5 Science

32.26%

18.64% 18.44%

30.66%

0

50

100

150


Frequency

Grade 8 Science


72

Someconcernsarenotedinmathematics,whererelativelyhigherpercentagesofstudentsarescoringatLevel1andveryfewatLevel2.However,thisfindingisconsistentwiththerangeofpossiblescores,whereLevel2insomecasesonlyhastwopossiblescalescorepoints(e.g.,Grade7,whereLevel2existsbetween207-208scaledscores).Theadditionof1-2lowcomplexityitemsperassessmentwillbeeffectedinmathematicstoaddressthisconcern,aswell.3.4ValidityBasedonRelationstoOtherVariablesPerhapsthebestmodelforunderstandingcriterion-relatedevidencecomesfromCampbellandFiske(1959)7intheirdescriptionofthemulti-trait,multi-methodanalysis[wetranslatetheterm‘trait’tomean‘skill’].Inthisprocess(several)differenttraitsaremeasuredusing(several)differentmethodstoprovideacorrelationmatrixthatshouldreflectspecificpatternssupportiveoftheclaimbeingmade(thatis,providepositivevalidationevidence).Sometimes,thesevariousmeasuresareofthesameorsimilarskills,abilities,ortraits,andothertimestheyareofdifferentskills,abilities,ortraits.Wepresentdatathatquiteconsistentlyreflecthigherrelationsamongitemswithinanacademicsubjectthanbetweenacademicsubjects.Wealsopresentdatainwhichperformanceonitemsistotaledwithincategoriesofdisability,expectingrelationsthatwouldreflect

7Campbell,D.T.,&Fiske,D.W.(1959).Convergentanddiscriminantvalidationbythemulti-trait,multi-

methodmatrix.InW.A.Mehrens&R.L.Ebel(Eds.),Principlesofeducationalandpsychologicalmeasurement:Abookofselectedreadings(pp.273-302).Chicago,IL:RandMcNally&Company.

18.86%

16.09%

24.74%

40.31%

0

50

100

150

200


Frequency

Grade 11 Science


73

appropriatedifferences(seeTindal,McDonald,Tedesco,Glasgow,Almond,Crawford,&Hollenbeck,2003).8

ConvergentandDivergentValidityDocumentationCriterionvalidityinformationisdifficulttodocumentwithAA-AAAS,asmostSWSCDdonotparticipateinanystandardizedassessmentoutsideoftheORExtand/orORorainOregon.DivergentvalidityevidenceisgarneredviacomparisonsofORExtresultstoORoraoutcomesshowsthatstudentswhoseORExtassessmentsarediscontinuedexhibitseriouslimitationsinattention,basicmathskills,andreceptiveandexpressivecommunicationskills.ThemedianORExtELAscoreforSWSCDwhoparticipatedintheORorawas4.0.ThemedianmathematicsORExtscorewas4.0,andthemedianscienceORExtscoreforSWSCDwhowereevaluatedwiththeORorawas0.0.PearsoncorrelationsbetweenthetotalrawscoresontheORExtandthetotalrawscoreontheORorawereconductedtoaddresstherelationshipbetweentotalperformanceoneachassessment.ThecorrelationbetweenELAandORorascoreswas0.56,betweenMathandORorascoreswas0.52,andbetweenScienceandORorascoreswas0.33.Asexpected,theORoraresultsprovidedivergentvalidityevidencefortheORExt.Wewouldnotexpectastrongrelationshipbetweenthescores,asstudentswhoseORExttestingisdiscontinuedaregenerallyunabletoaccesstheacademiccontentontheORExt,evenwiththerequisitereductionsindepth,breadth,andcomplexity.ConvergentevidencethattheORExtisassessingappropriateacademiccontentisprovidedbyQAandQTresponsestotheconsequentialvaliditysurvey.Respondentstothesurveygenerallyagreethat,“TheitemsintheOregonExtendedAssessmentaccuratelyreflecttheacademiccontent(whatthestudentshouldknow)thatmystudentswithsignificantcognitivedisabilitiesshouldbelearning,asdefinedbygradelevelcontentstandards(CCSS/NGSS)andtheEssentializedAssessmentFrameworks”(11%StronglyAgree&54%Agree).Inaddition,theyalsoagreedwiththestatementthat,“TheitemsintheOregonExtendedAssessment,whichprimarilyaskstudentstomatch,identify,orrecognizeacademiccontent,areappropriatebehaviorstoreviewtodeterminewhatmystudentswithsignificantcognitivedisabilitiesareabletodo”(18%StronglyAgree&64%Agree).TheconsequentialvalidityresultsdemonstratethattheORExtissamplingacademicdomainsthatthefieldofQAsandQTsdeemappropriateintheareaofacademics.

AnalysesWithinandAcrossSubjectAreasWeconductedcorrelationalanalysestofurtherexplorethevalidityoftheORExt.Wefirstdescribethepurposeoftheanalysis,aswellasouranticipatedresults.Wethendiscussourobservedresultsbeforeconcludingwithanoverallevaluativejudgmentofthevalidityofthetest.Inthecorrelationalanalysis,weexplorethecorrelationsamongstudents’totalscoresacrosssubjectareas.Thepurposeoftheanalysiswastoinvestigatehowstronglystudents’8Tindal,G.,McDonald,Tedesco,M.,Glasgow,A.,Almond,P.,Crawford,L.,&Hollenbeck,K.(2003).Alternate

assessmentsinreadingandmath:Developmentandvalidationforstudentswithsignificantdisabilities.ExceptionalChildren,69(4),481-494.


74

scoresinoneareawererelatedtostudents’scoresinothersubjectareas.Ifthecorrelationswereexceedinglyhigh(e.g.,above.90),itwouldindicatethatthescoreastudentreceivesinanindividualsubjecthaslesstodowiththeintendedconstruct(i.e.,reading)thanwithfactorsidiosyncratictothestudent.Forexample,ifallsubjectareascorrelatedat.95,thenitwouldprovidestrongevidencethatthetestswouldbemeasuringaglobalstudent-specificconstruct(i.e.,intelligence),andnottheindividualsubjectconstructs.Wewouldexpect,however,thatthetestswouldcorrelatequitestronglygiventhatthesamestudentswereassessedmultipletimes.Therefore,wewouldexpectmoderatelystrongcorrelations(e.g.,0.7)simplybecauseofthewithin-subjectdesign.Idiosyncraticvarianceassociatedwiththeindividualstudentisthuscaptured.

CorrelationalAnalysesResultsFullresultsofthePearson’sproduct-momentcorrelationanalysisbycontentareaandgradelevelarereportedbelow.Theresultsaresignificant,yettheoverallcorrelationsacrosscontentareassuggestthatweareindeedmeasuringdifferent,thoughstronglyrelatedconstructs,withbetween-testscaledscorecorrelationsrangingfrom0.81to0.89.Grade3ContentAreaCorrelationsn-size ContentArea ELARITCorrelation MathRITCorrelation

601 ELA – .85*Math –

*p<.001Grade4ContentAreaCorrelationsn-size ContentArea ELARITCorrelation MathRITCorrelation

583 ELA – .84*Math –

*p<.001Grade5ContentAreaCorrelationsn-size ContentArea ELARIT

CorrelationMathRITCorrelation

ScienceRITCorrelation

588ELA – .85* .85*Math – .81*Science –

*p<.001Grade6ContentAreaCorrelationsn-size ContentArea ELARITCorrelation MathRITCorrelation

519 ELA – .83*Math –

*p<.001


75

Grade7ContentAreaCorrelationsn-size ContentArea ELARITCorrelation MathRITCorrelation

508 ELA – .84*Math –




493ELA – .84* .89*Math – .83*Science –




572ELA – .83* .85*Math – .84*Science –

*p<.001ResultsofthePearson’sproduct-momentcorrelationanalysiswithinEnglishlanguagearts(ELA:Reading:Writing)arereportedbelowandsuggesthighcorrelationsbetweenELAandReading,asexpected,from.95to.97.WritingiscorrelatedwithELAfrom.91to.94andwithreadingfrom.79to.88.


76

EnglishLanguageArtsSubscoreCorrelationsn-size GradeScaled

ScoreELARITCorrelation

ReadingRITCorrelation

WritingRITCorrelation

6013ELA – .95* .91*3Reading – .80*3Writing –







*p<.001TheORExtassessmentsappeartobemeasuringseparateconstructs,asintended,indicatedbythecorrelations.Nounexpectedandconsistenttestfunctioningstatisticsarepresentbasedonstudentcharacteristicsthatshouldnotberelated,suchasgenderandethnicity.Studentperformanceappearstobeprimarilyrelatedtoitemdifficultyandnottheresultofconstructirrelevantaspectsthathavebeenreviewed.


77

CriticalElement4-TechnicalQuality:Other4.1ReliabilityTestreliabilitycanbeviewedthroughseverallenses,allofwhichdocumenthowconsistentlyanassessmentperformsacrossoccasions,contexts,andraters9.Typicalstrategiesforaddressingreliabilityincludedocumentationofinternalconsistency,split-halfreliability,andtest-retestreliability.Ifmultipleformsareimplemented,testformreliabilitydocumentationisalsorequisite.TheimplementationplanfortheORExtincludesinitialdocumentationofinternalconsistency(Cronbach'salpha).The2015-16technicalreportwillincludeinternalconsistencyestimates,split-halfreliabilityanalyses,aswellasasmalltest-retestassessmentofreliabilitycomparisonsbymeansofourpilottabletadministrationstudy.ThereisonlyonetestformfortheORExt,sotestformcomparisonsarenotpossible.

4.1ATestReliabilityMarginalreliabilityresults(truescorevariance/truescorevariance+errorvariance)demonstratethatthetestsarequitereliableatthetotaltestlevel.Fullreliabilitystatisticsforeachoftheoperationaltestsadministeredthisyearareprovidedbelow.Theseresultsdemonstratethatthetotaltestreliabilitieswerequitehigh,rangingfrom.87to.92.Eachtablebelowprovidesthecontentarea,grade,andthemarginalreliabilities.Alltestformswerecomposedof36operationaland12embeddedfield-testitems.EnglishLanguageArtsThetestreliabilitiesforELAwereinthehighrange,from.87to.92.

Grade MarginalReliability3 0.924 0.925 0.916 0.917 0.908 0.9111 0.87

MathematicsThetestreliabilitiesformathematicswereinthehighrange,from.88to.91.

Grade MarginalReliability3 0.914 0.915 0.906 0.90

9AmericanEducationalResearchAssociation(AERA),AmericanPsychologicalAssociation,&NationalCouncilonMeasurementinEducation(2014).Standardsforeducationalandpsychologicaltesting.Washington,DC:AERA


78

Grade MarginalReliability7 0.918 0.8811 0.90

ScienceThetestreliabilitiesforsciencewereinthehighrange,from.87to.91.

Grade MarginalReliability5 0.918 0.8811 0.87

TestInformationFunctionsThetestinformationfunctionspublishedbelowalsoindicatethatthescalesexhibitareliabilitygreaterthanorequalto.80forallproficient-levelcutscores.EnglishlanguageartsTIFs


79


80


81


82

MathematicsTIFs


83


84


85

ScienceTIFs


86


87

ValidationofORExtVerticalScalesThetestcharacteristiccurves(TCCs)forthegrade-levelassessmentsinELAandmathematicsdemonstrateincrementallyincreasinggrowthandtestdemandsacrossGrades3-8,withtheexceptionofGrade7mathematics.TheGrade7mathematicsassessmentwasrevisedtobemoredifficultlastyear,butclearlymoreelaborationofthiseffortisneededtoaddressitslocationontheTCC.Grade11andsciencetestsarenotverticallyscaled;TCCsarethusnotpresentedforGrade11orscience.AllRaschmodelscaling,aswellasthedatavisualizationsfortheTCCswereconductedintheRsoftware3.3.2environment(RCoreTeam,2016)usingther2Winstepspackage(Anderson,D.,2017).

-4 -2 0 2 4

010

2030

40

Θ

Exp

ecte

d To

tal R

aw S

core

Test Characteristic Curve

g3ELA.g4ELA.g5ELA.g6ELA.g7ELA.g8ELA.


88

-4 -2 0 2 4

010

2030

40

Θ

Exp

ecte

d To

tal R

aw S

core

Test Characteristic Curve

g3Math.g4Math.g5Math.g6Math.g7Math.g8Math.


89

4.1BOverallandConditionalStandardErrorsofMeasureTheaverageSEMassociatedwitheachcutscorefor2016-17studentdataarepresentedinthetablebelow,supportedbyaKEY.TheSEMsdecreasedinalmostallcasescomparedtolastyear,suggestingthatthemeasuresaremorereliablewhenstudenteligibilityismorestrictlycontrolled.SeeSection4.2belowformeansandstandarddeviationsbygradeandsubjectarea.SEM = StandardErrorofMeasureassociatedwiththecutscoretotheleft;averaged tothetenths'place.Level1 = DoesNotYetMeet(notincludedasthelowestlevelofproficiency)Level2 = NearlyMeetsLevel3 = MeetsLevel4 = ExceedsEnglishLanguageArtsGrade Level2 SEM Level3 SEM Level4 SEM AVG3 192 4.44 213 3.90 228 5.20 5.564 200 3.90 213 3.90 228 5.20 6.005 202 4.00 220 4.20 232 5.70 6.476 205 3.70 220 4.00 233 5.70 6.417 208 3.70 222 4.00 236 5.70 6.538 213 3.60 224 3.90 236 5.40 6.4611 899 3.80 920 4.30 927 5.40 8.41

MathematicsGrade Level2 SEM Level3 SEM Level4 SEM AVG3 192 3.90 201 3.80 218 5.10 5.104 193 3.80 206 3.80 219 5.00 4.965 193 4.20 206 3.80 220 4.40 5.306 204 3.60 208 3.60 222 4.50 4.957 207 3.80 209 3.90 223 5.30 5.338 208 3.70 212 3.70 226 4.40 4.9511 901 3.60 907 3.60 922 4.85 4.96

ScienceGrade Level2 SEM Level3 SEM Level4 SEM AVG5 506 3.70 517 4.10 530 5.60 6.648 810 3.80 820 4.40 831 6.10 7.3611 901 3.60 914 4.00 929 6.20 7.88

4.1CClassificationAccuracy&Consistency

Resultsfromthe2016-17ORExttestadministrationwereanalyzedusingRudner’sclassificationindex(Rudner,2005).Resultscloserto1.0indicatethelikelihoodthatastudentwasappropriatelyclassifiedasproficientornotproficient(accuracy)andthelikelihoodthatthestudentwouldbeclassifiedinthesamecategorygivenanadditionaltest


90

administration.Thecalculationutilizesitemdifficultyandthetavaluedistributions,aswellasrelatedstandarderrorsofmeasurement,togenerateprobabilisticestimatesbasedononetestadministration.Completeresults,generatedfromthecacIRTpackageinR,areprovidedbelow.Resultsdenoteveryhighlevelsofclassificationaccuracyandconsistency.TestClassificationAccuracyGrade EnglishLanguage

ArtsMathematics Science

3 0.94 0.92 -4 0.94 0.92 -5 0.95 0.91 0.956 0.95 0.92 -7 0.95 0.93 -8 0.94 0.88 0.9411 0.95 0.91 0.93TestClassificationConsistencyGrade EnglishLanguage

ArtsMathematics Science

3 0.91 0.89 -4 0.92 0.88 -5 0.92 0.87 0.926 0.93 0.88 -7 0.92 0.90 -8 0.92 0.84 0.9211 0.93 0.88 0.93TheORExtisnotacomputer-adaptiveinstrumentsoestimateprecisiondocumentationbaseduponthattestdesignisnotprovided.4.2FairnessandAccessibilityThestatehastakenstepstoensurefairnessinthedevelopmentoftheassessments,includingananalysisofeachtestitembyOregonteachersnotonlyforlinkagetostandards,butalsoforaccess,sensitivity,andbias(seeAppendix3.1A).Inaddition,wereviewedtestfunctioningasrelevanttorace/ethnicityanddisabilitysubgroups.Thisprocessincreasesthelikelihoodthatstudentsarereceivinginstructioninareasreflectedintheassessment,andalsothattheitemsarenotbiasedtowardaparticulardemographicorsub-group.

DifferentialItemFunctioningAnalysesToinvestigateDifferentialItemFunctioning(DIF),theMantel-Haenszeltestusinga

purificationprocesswasconducted(Holland&Thayer,1988;Kamata&Vaughn,2004)withtheRsoftwareusingthedifRpackage(Magisetal.,2013).WhenusingtheMantel-HaenszeltesttoinvestigateDIF,contingencytablesareconstructed,andtheresultingoddsforthefocalgroupansweringtheitemcorrectlyarecomparedtotheoddsforthereferencegroup.Givenn-sizelimitations(Scott,etal.,2009),wewereabletoconducttwoanalyses:


91

a)White/Non-Whiteandb)Male/Female.WhitesandMaleswerethefocalgroupsandNon-WhitesandFemaleswerethereferencegroups,respectively.Thecontingencytablesummarizescorrectandincorrectresponsestoeachitembyrespondents’totalrawscorebysubgroup(Kamata&Vaughn,2004).Ifthereisnodifferenceinperformanceforthetwogroups,theoddsratioofthefocalgroupperformancetoreferencegroupperformancewillequalone.Anoddsratiogreaterthanonemeansthefocalgroupisperformingbetterthanthereferencegroup,withtheoppositebeingtrueforoddsratioslessthanone.

ThedifRpackagecontainsabuiltinalgorithmtoconductpurificationautomatically,sowewereinterestedinhowthisalgorithmfunctionedrelativetotheiterationsconductedmanuallyusingSPSS.WeusedcriteriaoutlinedbytheEducationalTestingService(ETS)forDIFClassification(Holland&Thayer,1988)todeterminewhetherornotitemsexhibitedDIF,asthedifRpackagereportsdeltavaluesbydefault,definedas

∆!"= −2.35 ∗ 𝑙𝑛(𝛼!!) (1)where𝑙𝑛(𝛼!!) = 𝛽!"! .TheHollandandThayercriteriafor∆!" isasfollows:

∆!" ≤ 1.0=“A”items,1 ≤ ∆!" ≤ 1.5=“B”items,and ∆!" ≥ 1.5=“C”items.

TheHollandandThayercriteriawereusedforallMantel-Haenszelanalyses.Itemsthatwereflaggedas“C”levelitemswerereviewedbyBRTresearchersforpotentialbiases.Ifbiasesareidentified,theitemisremovedfromtheitempool.DIFanalyseswereperformedexpostfactoonthe2015-16ORExtoperationalitemstoaddresslongitudinaltrends.OnlythreeELAitemswereidentifiedasexhibitinga“C”levelDIFacrossboth2016and2017.ThosethreeELAitems,oneinGrade5thatexhibitedDIFthatprivilegedWhiteexaminees,oneinGrade4thatprivilegedFemaleexaminees,andoneinGrade8thatprivilegedFemaleexaminees,wereremovedandwillnotbeusedin2017-18orthereafter.DIFanalyseswillalsobeperformedinthe2017-18schoolyeartocontinuetoaddressDIFlongitudinally.Allitems,includingfieldtestitems,wereincludedintheanalyses.Thereareatotalof48itemsoneachassessment.


92

WithintheWhite/Non-Whiteanalysis,10outof18itemsflaggedas“C”levelitemsprivilegedNon-WhitetestparticipantsinELA,2outof5privilegedNon-WhitetestparticipantsinMathematics,and2outof7privilegedNon-WhitetestparticipantsinScience.Overall,DIFflaggingbasesonracewasrelativelybalanced,with14privilegingstudentswhowereNon-Whiteand16privilegingstudentswhowereWhite.White/Non-WhiteDIFAnalysesResultsContentArea

GradeNon-Whiten

Whiten

“C”ItemFrequency

ItemsRemovedCount

EnglishLanguageArts

3 248 359 3 04 263 320 2 05 247 348 1 16 232 292 7 07 208 302 3 08 202 304 0 N/A11 222 363 2 0

Mathematics 3 247 354 1 04 264 323 0 N/A5 248 349 0 N/A6 232 291 2 07 208 301 1 08 201 305 0 N/A11 222 363 1 0

Science 5 250 344 2 08 199 300 5 011 219 359 0 N/A


93

IntermsoftheMale/Femaleanalyses,10outof16itemsflaggedas“C”levelitemsprivilegedFemalesinELA,4outof9flaggeditemsprivilegedFemalesinMathematics,and8outof11flaggeditemsprivilegedFemalesinScience.Overall,DIFflaggingbasedonsexwasrelativelybalanced,with22privilegingFemalesand14privilegingMales.Male/FemaleDIFAnalysesResultsContentArea

Grade Femalen

Malen

“C”ItemFrequency

ItemsRemovedCount

EnglishLanguageArts

3 181 426 4 04 200 383 2 15 209 386 2 06 171 353 1 07 152 358 1 08 162 344 4 111 220 365 2 0

Mathematics 3 180 421 1 04 201 386 1 05 208 389 1 06 170 353 0 N/A7 152 357 4 08 162 344 1 011 221 364 1 0

Science 5 209 385 4 08 162 337 5 011 216 362 2 0

Race–EthnicityPercentagesandTotalsbyContentAreaandGradeLevel

ThefullethnicanddisabilitydemographicsforstudentstakingtheORExtarereportedbelow.Studentsethnicity/racewasreportedinsevencategories:(a)AmericanIndian/AlaskanNative,(b)Asian,(c)BlackorAfrican-American,(d)Multi-ethnic,(e)NativeHawaiianorOtherPacificIslander,(f)Hispanic,or(g)White.ThemajorityofstudentswerereportedasWhite(55-62%)orHispanic(22-29%).Theseresultsarelargelyconsistentwiththedemographicsreportedforthegeneralassessments,thoughpercentagestakingtheORExtareslightlyhigherformoststudentsofcolorandgenerallylowerforstudentswhoareAsianorWhite(seeAppendix4.2).


94

Englishlanguagearts GradeEthnicity-Race 3 4 5 6 7 8 11

AmericanIndian/AlaskaNative

.02 .02 .02 .03 .03 .02 .03

Asian .02 .04 .03 .03 .05 .04 .02BlackorAfrican-American

.03 .03 .02 .03 .03 .03 .05

Multi-ethnic .05 .07 .06 .08 .05 .06 .06NativeHawaiianorOtherPacificIslander

* * * * * * *

Hispanic .27 .29 .26 .27 .24 .23 .22White .59 .55 .59 .56 .60 .60 .62Totaln-sizes 584 571 574 507 504 490 563Note.*n<10;percentagesmaynotaddto1.0duetorounding.Mathematics GradeEthnicity-Race 3 4 5 6 7 8 11

AmericanIndian/AlaskaNative

.02 .02 .02 .03 .03 .02 .02

Asian .02 .04 .03 .03 .05 .04 .02BlackorAfrican-American

.03 .03 .02 .03 .03 .03 .05

Multi-ethnic .05 .07 .06 .08 .05 .06 .06NativeHawaiianorOtherPacificIslander

* * * * * * *

Hispanic .27 .29 .26 .27 .24 .23 .22White .59 .55 .59 .56 .60 .60 .62Totaln-sizes 578 575 575 506 502 490 563Note.*n<10;percentagesmaynotaddto1.0duetorounding.


95

Science GradeEthnicity-Race 5 8 11AmericanIndian/AlaskaNative

.02 .02 .03

Asian .03 .04 .02BlackorAfrican-American .03 .03 .05Multi-ethnic .06 .06 .06NativeHawaiianorOtherPacificIslander

* * *

Hispanic .27 .23 .22White .59 .60 .62Totaln-sizes 573 483 557Note.*n<10;percentagesmaynotaddto1.0duetorounding. StudentreportedexceptionalitiesincludedIntellectualDisability(ID),HearingImpairment(HI),VisualImpairment(VI),Deaf-Blindness(DB),CommunicationDisorder(CD),EmotionalDisturbance(ED),OrthopedicImpairment(OI),TraumaticBrainInjury(TBI),OtherHealthImpairment(OHI),AutismSpectrumDisorder(ASD),andSpecificLearningDisability(SLD).ThemajorityofstudentswhoparticipatedintheORExtwerestudentswithID(30-45%)andstudentswithASD(28-34%),followedbystudentswithOHI(11-16%).ODEpolicyfor2015-16changedtorequirestudentswhoparticipateintheORExttotaketheassessmentinallrelevantcontentareas.Thereisthusverylittlechangeintermsofparticipationpercentagesacrosscontentareas,asevidencedbythetotaln-sizespergradeleveldisplayedbelow.

ExceptionalityPercentagesByContentAreaandGradeLevelEnglishlanguagearts GradeCategory 3 4 5 6 7 8 11ID10 .30 .34 .37 .36 .39 .45 .43HI20 * * * * * * *VI40 * * * * * * *DB43 * * * * * * *CD50 .09 .08 .07 .06 .05 .03 .04ED60 .02 .02 .02 * .02 * .02OI70 .03 .03 .04 .03 .05 .03 .03TBI74 * * * * * * *OHI80 .14 .14 .16 .13 .12 .11 .11ASD82 .29 .29 .28 .34 .30 .31 .28SLD90 .06 .06 .05 .05 .06 .03 .06Totaln-sizes 584 571 574 507 504 490 563Note.*n<10;percentagesmaynotaddto1.0duetorounding.


96

Mathematics GradeCategory 3 4 5 6 7 8 11ID10 .30 .34 .37 .36 .39 .45 .43HI20 * * * * * * *VI40 * * * * * * *DB43 * * * * * * *CD50 .09 .08 .07 .06 .05 .03 .05ED60 .02 .02 .02 * .02 * .02OI70 .04 .03 .04 .03 .05 .03 .03TBI74 * * * * * * *OHI80 .16 .14 .16 .14 .12 .11 .11ASD82 .31 .29 .29 .34 .30 .31 .28SLD90 .06 .06 .05 .05 .06 .03 .05Totaln-sizes 578 575 575 506 502 490 563Note.*n<10;percentagesmaynotaddto1.0duetorounding.Science GradeCategory 5 8 11ID10 .36 .45 .42HI20 * * *VI40 * * *DB43 * * *CD50 .06 .03 .05ED60 .02 * .03OI70 .04 .04 .03TBI74 * * *OHI80 .16 .11 .11ASD82 .29 .31 .28SLD90 .05 .03 .05Totaln-sizes 573 483 557Note.*n<10;percentagesmaynotaddto1.0duetorounding.

ObservedMeansandStandardDeviationsThefollowingtablesprovideinformationregardingobservedmeansandstandarddeviationsbycontentareaandgradelevel.TheGrade3-8Englishlanguageartsandmathematicsscaledscoresarecenteredon200,whileallGrade11scoresarecenteredon900(toreinforcethattheyarenotontheverticalscale).Scienceiscenteredon500atGrade5andcenteredon800atGrade8.Theverticallyscaledscoresgenerallyconveyincrementalgainsinachievementacrossgradelevels,thoughtheresultssuggestsmalllossesappearingatGrade8inELA.Thesescaleswereselectedtoclearlydeterminewhetherscoresareonthesamescaleandalsotodifferentiateamongthestatewide


97

assessmentsinusetoavoidconfusion(i.e.,SBA,OAKS,ORExt,ELPA,KA).ThegeneralpatternisthatRITscoresdecreasedfrom2014-15to2015-16.Thisdecreaseisattributednottothescale,nortodecelerationofgrowth,buttothesubstantiveshiftinthetestedstudentpopulationasaresultofODEeligibilityguidelines.Thescalefrom2015-16to2016-17appearstohavestabilizedbecausethestudentpopulationtestedwasmoreconsistent.2014-15RITScores

ELA Math Science

Grade Mean SD Mean SD Mean SD3 219.3 24.6 201.5 20.8

4 222.8 23.6 204.8 19.8 5 224.9 25.0 205.3 18.1 517.6 25.6

6 226.3 24.0 207.7 17.7 7 226.4 25.0 207.9 19.0 8 225.4 24.1 207.8 17.3 822.1 25.8

11 922.5 28.5 903.8 21.1 920.8 27.7

2015-16RITScores

ELA Math Science

Grade Mean SD Mean SD Mean SD3 210.3 23.0 197.6 20.2

4 212.3 22.9 198.1 18.7 5 217.1 24.5 201.2 17.2 514.2 22.1

6 220.1 25.5 204.8 17.6 7 223.6 28.9 205.4 19.0 8 221.2 24.8 206.7 17.2 819.0 25.6

11 920.7 27.7 902.3 20.0 918.0 24.92016-17RITScores

ELA Math Science

Grade Mean SD Mean SD Mean SD3 209.64 21.73 196.16 18.96 - -4 213.13 23.38 198.45 17.98 - -5 213.85 25.01 198.37 19.54 513.65 24.596 216.65 23.76 203.29 17.43 - -7 220.53 23.88 205.13 19.87 - -8 219.48 24.28 205.92 16.26 817.96 24.3611 922.05 26.37 903.07 17.57 919.41 24.25


98

ObservedMeansReportedbySexThefollowingtablesprovideinformationregardingaveragestudentperformancebygradelevelandsex(Female/Male)ineachofthecontentareasassessedontheORExt.SignificantdifferencesbasedonaWelchtwosamplet-testarenotedinGrade4ELA,Grades4,5,and8inmathematicsandGrades5and8inscience.EnglishLanguageArts GradeSex 3 4 5 6 7 8 11Female 209.65 210.30* 211.85 216.44 219.46 218.88 921.56Male 209.56 214.71* 214.70 217.55 220.99 220.33 922.39Note.*p<.05Mathematics GradeSex 3 4 5 6 7 8 11Female 193.31 194.16* 195.27* 201.91 202.63 203.67* 901.87Male 196.73 200.76* 199.87* 204.60 206.41 207.31* 904.05Note.*p<.05Science GradeSex 5 8 11Female 510.36* 814.84* 916.67Male 515.00* 820.02* 920.88Note.*p<.05

ObservedMeansReportedbyRaceThefollowingtablesprovideinformationregardingaveragestudentperformancebygradelevelandrace(Non-White/White)ineachofthecontentareasassessedontheORExt.Significantdifferencesarenotedbytwosamplet-testsinELAGrade3and8andinGrade8inScience.EnglishLanguageArts GradeRace 3 4 5 6 7 8 11Non-White 207.45* 212.28 212.65 217.34 220.35 216.95* 921.01White 211.06* 213.91 214.41 217.07 220.67 221.81* 922.73Note.*p<.05


99

Mathematics GradeRace 3 4 5 6 7 8 11Non-White 194.57 197.63 197.71 203.73 205.18 205.82 903.24White 197.06 199.15 198.62 203.73 205.37 206.40 903.24Note.*p<.05Science GradeRace 5 8 11Non-White 511.42 815.74* 917.79White 514.71 820.10* 920.86Note.*p<.05

ObservedMeansReportedbyExceptionalityStatusThefollowingtablesprovideinformationregardingaveragestudentperformancebygradelevelandexceptionalitycategoryineachofthecontentareasassessedontheORExt.StudentswithSLDweregenerallythehighestperforminggroup,thoughstudentswithCDandEDperformedhigheratcertaingradelevels/contentareas.ThelowestperforminggroupwasconsistentlystudentswithOI,followedbystudentswithIDorASD,dependingupongradelevel.EnglishLanguageArtsExceptionality GradeCode 3 4 5 6 7 8 11ID10 209.12 210.70 209.76 214.33 219.82 219.62 916.96HI20 * * * * * * *VI40 * * * * * * *DB43 * * * * * * *CD50 216.82 222.02 229.63 230.13 234.08 232.88 939.00ED60 226.38 222.21 235.29 236.17 229.80 * 941.36OI70 185.50 181.22 186.52 196.93 187.09 198.24 911.32TBI74 * * * * * * *OHI80 212.31 218.47 213.51 220.28 221.47 226.15 935.03ASD82 206.11 210.82 214.81 214.46 219.90 216.66 918.10SLD90 228.34 228.60 238.22 237.22 236.03 241.06 942.35Note.*n<10;theGrade11scaleisuniqueinELAandnottobecomparedtoGrades3-8.


100

MathematicsExceptionality GradeCode 3 4 5 6 7 8 11ID10 194.91 196.47 195.61 201.23 204.41 205.61 900.20HI20 * * * * * * *VI40 * * * * * * *DB43 * * * * * * *CD50 201.28 207.77 211.11 213.63 218.92 214.88 915.00ED60 213.15 210.43 214.36 212.00 211.20 * 916.07OI70 176.30 173.11 178.10 188.50 178.22 192.06 894.89TBI74 * * * * * * *OHI80 197.45 202.46 198.53 204.93 208.92 209.85 910.87ASD82 193.71 194.81 198.68 203.02 203.37 204.30 901.01SLD90 214.74 215.54 214.00 218.52 219.71 219.06 915.73Note.*n<10;theGrade11scaleisuniqueinMathandnottobecomparedtoGrades3-8.ScienceExceptionality GradeCode 5 8 11ID10 509.99 818.50 915.83HI20 * * *VI40 * * *DB43 * * *CD50 529.68 828.00 936.50ED60 539.79 * 943.79OI70 489.43 798.12 908.89TBI74 * * *OHI80 517.01 826.71 932.69ASD82 510.60 813.41 914.13SLD90 539.46 839.00 938.00Note.*n<10;allGradelevelscalesareuniqueinScienceandnottobecompared.

GraphsofObservedMeansByDisabilityThegraphsbelowconveyinformationsimilartothatsharedaboveingraphicform.Thegraphicsinclude95%confidenceintervalerrorbars,sodeterminingwhichsubgroupsperformedinamannerthatissignificantlybetterthanothersisreadilyapparentbylookingatthelocationoftheerrorbars.Errorbarsthatdonotoverlapintermsofthey-scalearesignificantlydifferent.Onlystudentswhogenerallyhadmorethan10membersateachgradelevelarereported.ThisrequiredtheremovalofgraphsforstudentsintheHI,VI,DB,andTBIcategories.


101

StudentswithOIareagainthelowestperforminggroup,beingsignificantlyoutperformedbyallothersubgroups.StudentswithSLDareconsistentlyoutperformingmostpeers,withstudentswithEDandCDperformingatsimilarlyhighlevels.StudentswithOIareconsistentlythelowestperforminggroup,whichledtoconcernsregardingtestaccessibility.However,theresultsoflastyear’sconsequentialvaliditystudydemonstratedthattheOIlabelisinsufficienttofullydescribetheseverityandrangeofconcomitantdisabilitiesthatstudentswhoseprimarylabelisOIconveys.

180

200

220

240

3 4 5 6 7 8Grade

ELA

Mea

n

ExceptionalityID

CD

ED

OI

OHI

ASD

SLD

Average ELA RIT Scores By Exceptionality


102

900

910

920

930

940

950

Grade 11

ELA

Mea

n

ExceptionalityID

CD

ED

OI

OHI

ASD

SLD

Average ELA RIT Scores By Exceptionality


103

160

180

200

220

3 4 5 6 7 8Grade

Mat

h M

ean

ExceptionalityID

CD

ED

OI

OHI

ASD

SLD

Average Math RIT Scores By Exceptionality


104

890

900

910

920

Grade 11

Mat

h M

ean

ExceptionalityID

CD

ED

OI

OHI

ASD

SLD

Average Math RIT Scores By Exceptionality


105

490

510

530

Grade 5

Sci

ence

Mea

n

ExceptionalityID

CD

ED

OI

OHI

ASD

SLD

Average Science RIT Scores By Exceptionality


106

780

800

820

840

Grade 8

Sci

ence

Mea

n

ExceptionalityID

CD

ED

OI

OHI

ASD

SLD



107

Grades3-8ObservedandUnconditionalGrowthExpectationsTheORExtwasredesignedin2014-15tosupportgrowthdeterminationsinGrades3-8inEnglishlanguageartsandmathematics.Averticalscaleusingabalanceddesignwasusedtodeveloptheinitialscale.Nowthatweareinthethirdyearofadministration,itbecamepossibletomodelgrowthexpectationsforELAandMathforSWSCDwhotooktheORExt.ThefollowinggraphsconveytheaveragegrowthexpectationsforSWSCDinOregonandshouldprovidesomecontextforunderstandingtypicalperformanceandaveragegrowthinIndividualizedEducationProgram(IEP)meetings.TheODEchangedtheeligibilitycriteriaforSWSCDtoparticipateintheORExtinthe2015-16schoolyear.Thishadanimpactonthetestedpopulation,astheexpectationsweremore

900

910

920

930

940

950

Grade 11

Sci

ence

Mea

n

ExceptionalityID

CD

ED

OI

OHI

ASD

SLD



108

prescriptive,andstudentpopulationsdecreasedbyanaverageof40%ineachcontentareaandgradeleveltested.ThischangealsoaffectedORExttestresults,asthestudentswhoparticipatedinthefirstadministrationbutnotinsubsequentadministrationsweregenerallyveryhighachieving.TogenerategrowthestimatesthatmatchedtheintendedstudentpopulationfortheORExt,namelystudentswhodidnotexittheassessmentafterthe2015administration,alldatasetsforgrowthmodelingexcludedthegroupofstudentswhoparticipatedinonlythe2015administration.Studentswhosegradeleveladvancementwasnottypicalwerealsoexcluded(n=18exclusionsinELAandmath,respectively).Allotherparticipantsweremaintained.Theobservedcohortmeansarerepresentedbelowforcomparisonpurposes.InELA,thescoresatGrade3averageaRITscoreof205.72.ByGrade8,theaverageRITscoreinELAis218.99.Intermsofobservedmeans,studentsthusgrowatotalof13.27RITscorepointsfromGrades3to8inELA,foranaverageannualgrowthrateof2.21RITscorepointsperyear.Inmathematics,theaverageGrade3RITscorewas193.20.ByGrade8,theaveragescorewas205.78.Students’observedmeansthusincreasedby12.58RITscorepoints,foranaverageannualgrowthrateof2.10RITscorepointsperyear.EnglishLanguageArtsObservedMeans2015–2017byCohort

Cohort1 Cohort2 Cohort3 Cohort4Grades 3-4-5 4-5-6 5-6-7 6-7-8

3 205.72 4 209.74 210.37 5 211.40 213.95 214.71 6 214.44 217.03 215.067 218.97 218.668 218.99

MathematicsObservedMeans2015-2017byCohort

Cohort1 Cohort2 Cohort3 Cohort4Grades 3-4-5 4-5-6 5-6-7 6-7-8

3 193.20 4 196.22 196.22 5 196.74 199.47 200.43 6 202.18 203.05 200.707 204.21 202.718 205.78

Observedmeanshideasubstantialamountofinformation,however,astheydonotaccountforthevarianceinscoresthatexistsinthepopulation.Wethusconductedunconditionalgrowthmodelstoparseoutthevarianceassociatedwitheachinterceptandslopeestimate.Weincludedmultiplecohortstoaddresstheobservednon-linearityinthegrowthestimates.AlldatapreparationandanalyseswereconductedintheRsoftware3.3.2environment(RCoreTeam,2016)usingthelme4package(Bates,Maechler,Bolker,&Walker,2015).Inaddition,thedatavisualizationsbelowwereconductedwithggplotinthe


109

tidyversepackage(Wickham,H.,2017).Cohorteffectswereaddressedbyaveragingacrossoverlappinggrades;however,theprocessofaveragingovercohortsshouldcontinueannually.UnconditionalModel-PredictedELAMeans2015–2017byCohort

Cohort1 Cohort2 Cohort3 Cohort4 Grades 3-4-5 4-5-6 5-6-7 6-7-8 AVG

SlopeEstimate 2.73 1.93 1.93 1.81 2.103 206.21 206.214 208.94 210.98 209.965 211.67 212.91 214.95 213.186 214.84 216.88 215.76 215.837 218.81 217.57 218.198 219.38 219.38

3 4 5 6 7 8

200

205

210

215

220

English Language Arts Growth

Grade

RIT

Sco

re

Cohort 1Cohort 2Cohort 3Cohort 4Average


110

UnconditionalModel-PredictedMathematicsMeans2015–2017byCohort Cohort1 Cohort2 Cohort3 Cohort4

Grades 3-4-5 4-5-6 5-6-7 6-7-8 AVGSlopeEstimate 1.66 2.85 1.78 2.47 2.19

3 193.72 193.724 195.38 196.42 195.905 197.04 199.27 200.77 199.036 202.12 202.55 200.57 201.757 204.33 203.04 203.698 205.51 205.51

3 4 5 6 7 8

190

195

200

205

210

Mathematics Growth

Grade

RIT

Sco

re

Cohort 1Cohort 2Cohort 3Cohort 4Average


111

Theunconditionalgrowthestimatesshowthattherewereinterestingcohorteffects,withCohort3averyhighachievingcohortinbothELAandmathematics.ThesecohorteffectsareworthyoffurtherstudyandimplythatcautionshouldbeusedwheninterpretinggrowthestimatesfortheORExtforspecificapplications.Whenaveragingacrosscohorts,studentsinELAachievedaRITscoreof206.21pointsinGrade3andgrewtoaRITscoreof219.38byGrade8.Theaveragegrowthwas2.10RITscorepointsperyear.Whenaveragingacrosscohorts,studentsinMathachievedaRITscoreof193.72pointsinGrade3andgrewtoaRITscoreof205.51bytheGrade8.Theaveragegrowthwas2.19RITscorepointsperyear.CurvilinearityisnotedintheELAdata,however,withmoregrowthoccurringattheearliergradesthanatthelatergrades.Mathematicsgrowthappearstobemorelinear.ORoraChangeScoresfrom2016to2017TheORoratotalrawscoresfrom2016and2017werecomparedtodeterminehowmuchchangewasexhibitedfromthefirstadministrationoftheORorain2016tothesecondadministrationin2017.Atotalof849studentsparticipatedintheORorain2016andatotalof772participatedin2017.Only473ofthosestudentsparticipatedintheORoraforbothyearsoftheadministration.Then-sizefortheplotsbelowincludesthose473students.TherangeofpossiblescoresontheORoraisfrom20to80.Themeanscorein2016was46.12,whilein2017themeanwas48.08.Theaveragechangefrom2016to2017ontheORorawas1.827points,buttherewasgreatvariationinchangescores(min=-60,max=+40).

2016 ORora Results

ORora Total Score

Frequency

20 30 40 50 60 70 80

010

2030

40

Mean Score (46.12)


112

2017 ORora Results

ORora Total Score

Frequency

20 30 40 50 60 70 80

010

2030

40

Mean Score (48.08)

ORora Change Scores 2016 to 2017

Delta

Frequency

-60 -40 -20 0 20 40

020

4060

80100

Average Change Score (1.827)


113

4.3FullPerformanceContinuumTheORExtisdesignedtosampletheCommonCoreStateStandardsinEnglishlanguagearts(Reading,Writing,andLanguage)andMathematics,aswellastheOregonScienceStandardsandNextGenerationScienceStandardsinscienceinapurposive,validatedmanner.TheORExttestblueprintsconveythebalanceofrepresentationexhibitedbytheassessment(seeAppendix2.1B).ThesetestblueprintsaresupportedbytheORExtExtendedAssessmentFrameworks(http://www.brtprojects.org/publications/training-modules),whichdefinetheassessablecontentontheORExtthathasbeenreducedindepth,breadth,andcomplexity(RDBC)usingourdefinedprocess(seeAppendix2.3A.3).Thedecisionsregardingwhichstandardstotargetforessentialization,aswellasthestrengthoflinkagebetweentheEssentializedStandardsandtheCCSS/ORSci/NGSShasbeenvalidatedbyOregonteachers,aswell(seeAppendix3.1A).Thoughasimplifiedandstandardizedapproachwastakentodesignitems,andefficiencyandaccesstotheassessmentincreasedforthemajorityofstudents(asevidencedbythedecreasedpercentagesofzeroscoresacrossallcontentareas),asmallsubgroupofstudentsremainswhocannotaccessanacademicassessment.ThisistrueeventhoughitemshavebeensignificantlyRDBCatthreelevelsofcomplexity(low-medium-highdifficulty).Asaresponse,ODEcommissionedBRTtodesignandimplementanobservationalratingscaleforthisgroupofverylow-performingstudents,calledtheOregonObservationalRatingAssessment(ORora)forthespring2016administration.TheORoratargetscommunication(expressiveandreceptive)andbasicskills(attention/jointattentionandmathematics)andprovidesdocumentationofstudentprogressoutsideofourclearlydefinedacademicdomains.Itemsonallassessmentswerescoredona2-pointscale,with1pointawardedforacorrectresponseand0pointsawardedforanincorrectresponse.Plotsareprovidedbelowforeachcontentareaandgradelevel,includingthepersonabilityanditemdifficultydistributions.Ingeneral,thedescriptivestatisticssuggestthatthetesthadanappropriaterangeofitemdifficultiesrepresented,fromeasytodifficult,withitemdifficultiesgenerallyrangingfrom-4.0to+4.0ontheRaschscale.TheassessmentsperformedasexpectedacrossallgradesandcontentareaswiththeexceptionofGrade7mathematics,asnotedabove.TheitempersondistributionsprovidedbelowdemonstratethattheORExtisprovidingaperformancecontinuumforstudentswhoparticipate.


114

EnglishLanguageArtsPerson/ItemDistributions


115


116


117

MathematicsPerson/ItemDistributions


118


119


120


121

SciencePerson/ItemDistributions


122

PersonAbilityandItemDifficultyTablesEnglishlanguagearts PersonAbility ItemDifficulty

Grade Mean SD Mean SD3 0.96 0.56 0.41 0.114 1.30 0.60 0.54 0.125 1.38 0.65 0.57 0.126 1.66 0.64 0.80 0.137 2.06 0.65 0.96 0.138 1.94 0.65 1.20 0.1311 2.20 0.84 0.41 0.11


123

Mathematics PersonAbility ItemDifficulty

Grade Mean SD Mean SD3 -0.38 0.51 -0.41 0.114 -0.15 0.50 -0.23 0.115 -0.16 0.53 0.25 0.116 0.34 0.49 0.58 0.117 0.51 0.53 0.05 0.128 0.60 0.49 1.02 0.1111 0.31 0.49 0.23 0.10Science PersonAbility ItemDifficulty

Grade Mean SD Mean SD5 1.37 0.66 0.60 0.128 1.80 0.74 0.53 0.1411 1.94 0.79 0.33 0.134.4ScoringAllscoringexpectationsfortheORExtareestablishedwithintheAdministrationManual(seeAppendix2.3,p.14).ThescoringproceduresforthenewORExthavebeensimplified,withstudentsreceivinga0foranincorrectresponseora1foracorrectresponse.InputfromthefieldgatheredfromConsequentialValiditystudiesdemonstratesthattheassessmentscoringproceduresaremuchmoreclearandeasiertoimplementthanpriorscoringapproaches(seeAppendix2.3B.10).BRTwasalsocommissionedtodevelopascaledscoreinterpretationguide,whichdescribesspecificstrategiesforinterpretingstudenttestscoresandsub-testscoresinReadingandWriting,andAchievementLevelDescriptors(ALDs)publishedwithintheIndividualStudentReports(seeAppendix6.4C)forannualperformance,growth,andaspartofEssentialSkillsrequirementsforverylowperformingstudents(seeAppendix2.1A).4.5MultipleAssessmentFormsTheORExtwasadministeredinonlyonegradelevelformforthe2016-17schoolyear,with36operationalitemsarrangedinorderofempiricaldifficultyand12embeddedfieldtestitems. 4.6MultipleVersionsofAnAssessmentTheORExtisprovidedinthestandardformat,butisalsoavailableinLargePrintandBrailledformats.Testcontentisidenticalacrossallthreeversions,withanoccasionalitembeingeliminatedontheBrailleversionduetoinaccessibility.Theseitemsdonotcountfororagainstthestudentinreporting.Substantivetestcomparabilityanalysesarenotfeasible,giventhesmalln-sizesofthesamplesinvolvedinthealternativeversions.


124

4.7TechnicalAnalysesandOngoingMaintenanceTheORExttechnicalanalysesthatdocumentreliabilityandvalidityareincludedinthistechnicalreport(seeSections3and4,respectively).ODEandBRTstaffreviewstheseanalysesannually.Necessaryadjustmentstotheassessmentaredeterminedpriortoimplementationofthesubsequentyear'sworkplan,whichelaboratestheareasofimprovementaswellasaspectsofthetestingprogramthatwillbemaintained.Thisdecision-makingissupportedbyinputfromthefieldgatheredfromtheConsequentialValiditystudy(seeAppendix2.3B.10).Onenoteworthyexampleoftheimpactofoursystemofongoingimprovementthisyearisthedevelopmentofadditionalcurricularandinstructionalresources,whichaddressesanareaofconcernexpressedbystakeholders.ThetrainingmoduleswedevelopedtoconnecttheassessmentresultsgarneredfromtheORExtandORorawithcurricularresourcesandinstructionalstrategiesthatarealigned.


125

CriticalElement5–InclusionofAllStudents

5.1ProceduresforIncludingSWDsTheOregonassessmentsystemprovidesexplicitguidanceregardingtheparticipationofallpublicschoolstudentsinitsstatewideassessmentprogram(seeSection1.4).

5.1AClearExplanationsoftheDifferencesBetweenAssessments

TheassessmentoptionsforallpublicschoolstudentsinOregonareelaboratedintheOregonTestAdministrationManual(seeAppendix1.4.2,p.7).TheseoptionsincludetheSmarterBalancedAssessmentinEnglishlanguageartsandmathematicsinGrades3-8&11,theOregonAssessmentofKnowledgeandSkillsinscienceinGrades5,8,&11,andinthesamecontentareasandgradelevelsforSWSCDwhotaketheORExt(seeAppendix1.4.2,p.92-93).SocialstudiesassessmentisadistrictoptionwithintheOAKSportal,aswell.Inaddition,expectationsfortheEnglishLanguageProficiencyAssessment(ELPA)andtheKindergartenAssessmentareprovided.

5.1BEligibilityDecisionsMadebyIEPTeamsAstudent'sIEPteamdetermineshowastudentwithdisabilitieswillparticipateintheOregonStatewideAssessmentprogram.TheIEPteammustaddresstheeligibilitycriteriaforparticipationintheORExtbeforedeterminingthattheassessmentistheappropriateoption(seeAppendix5.1B).

5.1CGuidelinesforAssessmentSelectionAsnotedearlier,IEPteamsmakedecisionsregardinghowstudentswithdisabilitiesparticipateintheOregonstatewideassessmentprogram.Atpresent,studentsparticipateinoneofthreeoptions:(a)studenttakesthegeneralassessmentwithorwithoutuniversaltools.(b)studenttakesthegeneralassessmentwithdesignatedsupportsand/oraccommodations,or(c)studenttakestheORExt.Guidelinesformakinguniversalsupport,designatedsupport,andaccommodationsdecisionsforthegeneralassessmentsareprovidedinAppendix2.3A.1.GuidelinesformakingthesedeterminationsforSWSCDwhoparticipateinAA-AAASareprovidedinAppendix5.1B.

5.1DInformationonAccessibilityOptionsInformationregardingaccessibilityoptionsforthegeneralassessmentcanbefoundwiththegeneralassessmentPeerReviewevidence.FortheORExt,accessibilityistreatedholistically,withuniversaldesignforassessmentconceptsembeddedintheitemdesignandawidevarietyofaccommodationsalsoavailableifneeded.Itemsarecraftedtobevisuallysimpleandclean.Graphicsupports,whicharealwaysblack/whitelinedrawings,areembeddedinallitemsatthelowlevelofcomplexitybutarephasedoutasitemsbecomemorecomplex.Itemsaredesignedtoincorporatesimplifiedlanguageunlessspecificacademicvocabularyandconceptsiswhatisbeingtested(seeAppendix2.3A.3).TheitemsontheORExtareallselectedresponse,withthreeresponseoptionsallowingformultiplemodesofaccess(e.g.,sayingtheanswer,pointingtotheanswer,eyegaze,switch,etc.).Alltextpresentedtostudentsisatleast18-ptfont(larger,ofcourse,inthelargeprintversion).


126

SampleitemsarepresentedinAppendix2.2.3.Allaccessibilitysupports,designatedsupports,andaccommodationsfortheORExtarepublishedinAppendix2.3A.1,p.36-43.ForstudentswhohaveverylimitedtonocommunicationandareunabletoaccesseventhemostaccessibleitemsontheORExt,anOregonObservationalRatingAssessment(ORora)wasimplementedin2015-16.TheORoraiscompletedbyteachersanddocumentsthestudent'slevelofcommunicationcomplexity(expressiveandreceptive),aswellaslevelofindependenceinthedomainsofattention/jointattentionandmathematics.Theadministrationinstructionsand2015-16resultsfortheORoraareincludedinAppendix5.1D.

5.1EGuidanceRegardingAppropriateAccommodationsGuidanceregardingappropriateaccommodationsispublishedinAppendix2.3A.1.DistrictandSchoolTestCoordinatorsprovideannualtrainingontestsecurityandadministration.TheORExtapproachesaccessaspartoftestdesign,asnotedaboveinSection5.1D.ThecomplexityofSWSCDcommunicationsystemsdemandssuchanapproach.Inaddition,comprehensiveaccommodationsareallowedinordertodecreasethechancesthatadisabilitymayinterferewithourabilitytomeasurethestudent'sknowledgeandskills.

5.1FAllSWDsEligiblefortheORExtODE'seligibilityguidelinesmakeitclearthatallSWDsareeligiblefortheORExt,regardlessofdisabilitycategory,andthatspecificdisabilitycategorymembershipshouldnotbeadeterminingfactorforconsideringparticipation(seeAppendix5.1B).

5.1GParentsInformedofAA-AAASConsequencesTheParentFAQsectionoftheGeneralAdministrationManualmakesitclearthatparentsmustbeinformedofthepotentialconsequencesofhavingtheirchildassessedagainstalternateachievementstandards,includingdiplomaoptions.Parentsarealsoinformedthatalternateachievementstandardsaredesignedtoreflectasignificantreductionindepth,breadth,andcomplexityandarethereforenotcomparabletogeneralacademicachievementstandards(seeAppendix2.3,p.28-32).

5.1HStateEnsuresORExtPromotesAccesstotheGeneralEducationCurriculumTheORExtisstronglylinkedtotheCCSS/ORSci/NGSS,asevidencedbyourlinkagestudyresults(seeAppendix3.1A).Theclaimisbasedonthefollowingwarrants:(a)ORExtitemsarealignedtotheEssentializedStandards;(b)theEssentializedStandardsarestronglylinkedtothegradelevelcontentstandards;therefore(c)theORExtitemsarestronglylinkedtogradelevelcontentexpectations.ItisthusexpectedthattheORExtpromotesaccesstothegeneraleducationcurriculumbyassessinggeneraleducationcontentthathasbeenreducedindepth,breadth,andcomplexityyetmaintainsthehighestpossiblestandardforSWSCD.Inaddition,ODEcommissionedBRTtoworkwithOregonteachersofSWSCDinthe2015-16schoolyeartodevelopavarietyofcurricularandinstructionalresourcesthatarealignedtotheEssentializedStandards.Theseresourcesinclude:(a)curriculartemplates,


127

(b)videotutorials,and(c)supportingdocumentsthatprovidespecificguidanceregardinghowtodeveloplessonplans,PresentLevelsofAcademicandFunctionalPerformance(PLAAFP)statements,andIndividualizedEducationProgram(IEP)goalsandobjectivesthatarealignedwiththeEssentializedStandards.Itisalsoexpectedthattheessentializationprocesswillgeneralizetomanystudentswhoareperformingoffgradelevel,notmerelytoSWSCD.AllresourcesarepublishedonaBRT-sponsoredwebsiteathttp://lms.brtprojects.org.5.2A–5.2CProceduresforIncludingELsInadditiontotheprogrammaticguidanceprovidedinAppendix1.4A.1relatedtoELprogrameligibilityandservices,ODEalsoprovidesguidancerelevanttotheinclusionofELsinthestatewideassessmentprograminAppendix1.4.2.ThoughtheORExtiscurrentlypublishedinEnglish,anappropriatelyqualifiedinterpretercanprovidetheassessmenttoanySWSCDfromdiverselanguagebackgrounds,includingAmericanSignLanguage.ODEhasdevelopedatrainingmoduletoincreasethestandardizationofASLadministrationforitsstatewideassessments,availableathttp://lms.brtprojects.org.AdditionalinformationregardingtheinclusionofELsinOregon'sgeneralassessmentsisprovidedinthegeneralassessmentPeerReviewevidence.5.3AccommodationsAllstatewideaccommodationguidanceispublishedintheAccessibilityManual(seeAppendix2.3A.1),outliningtheuniversaltoolsanddesignatedsupportsavailabletoallstudents,andaccommodations,availableonlytostudentswithdisabilitiesorstudentsservedbySection504Plans.Inaddition,themanualdefinesthesupportsasembedded,wheretheyareprovidedbytheonlinetestengine(e.g.,calculator,text-to-speech),ornon-embedded,wheretheymustbeprovidedbyaqualifiedassessor(e.g.,readaloud,scribe).Themanualalsomakesitclearthatthesesupportsarecontent-areaspecific,asauniversaltoolinonecontentareamaybeanaccommodationinanother.

5.3AAppropriateAccommodationsareAvailableforSWD/Section504AppropriateaccommodationsfortheORExtarepublishedinAppendix2.3A.1,p.36-43.Additionalaccommodationsforallstatewideassessmentsarealsopublishedinthismanual.TheOregonAccommodationsPanelreviewstheappropriatenessofthesupportslistedannually.Practitionersmayalsorequesttheadditionofanaccommodationthroughaformalprocess(seeAppendixE:ApprovalProcessforNewAccessibilitySupportswithinthemanual,Appendix2.3A.1,p.100-102).

5.3BAppropriateAccommodationsareAvailableforELsAsnotedinSections5.2A-C,theORExtisaccessibleinanycommunicationmodalitythroughtheuseofaninterpreter.AppropriateaccommodationsfortheORExtarepublishedinAppendix2.3A.1,p.36-43.Additionalaccommodationsforallstatewideassessmentsarealsopublishedinthismanual.TheOregonAccommodationsPanelreviewstheappropriatenessofthesupportslistedannually.Practitionersmayalsorequestthe


128

additionofanaccommodationthroughaformalprocess(seeAppendixE:ApprovalProcessforNewAccessibilitySupportswithinthemanual,Appendix2.3A.1,p.100-102).

5.3CAccommodationsareAppropriateandEffectiveInadditiontotheevidencegatheredduringthelinkagestudy(seeAppendix3.1A),whichsuggeststhattheORExtitemswereaccessibleandfreeofbiasevenbeforefinalediting,theappropriatenessofthesupportslistedinAppendix2.3A.1isreviewedannuallybytheOregonAccommodationsPanel.Practitionersmayalsorequesttheadditionofanaccommodationthroughaformalprocess(seeAppendixE:ApprovalProcessforNewAccessibilitySupportswithinthemanual,Appendix2.3A.1,p.100-102).ODEiscollectingaccommodationscodesfortheORExtfromQualifiedAssessorswhoopttoenterthisinformationinordertomakeperformancecomparisonsfeasible.Itishopedthatthisprocesswillberequiredbyspring2018.Theconsequentialvaliditystudyfor2018willincludequestionsregardingtheappropriatenessoftheavailableaccommodations,aswell.

5.3DAccommodationsareAppropriateandEffectiveODEhasaformalprocessstakeholderscanusetorequestaccommodationsthatarenotalreadypublishedintheAccessibilityManual(seeAppendixE:ApprovalProcessforNewAccessibilitySupportswithinthemanual,Appendix2.3A.1,p.100-102).5.4A–5.4EMonitoringTestAdministrationforSpecialPopulationsODEmonitoringoftestadministrationinitsdistrictsandschoolsiselaboratedwithinthegeneralassessmentPeerReviewevidenceandisthereforenotaddressedhere.


129

CriticalElement6–AcademicAchievementStandardsandreporting6.1StateAdoptionofAlternateAcademicAchievementStandardsforSWSCDTheOregonExtendedassessment(ORExt),Oregon'sAlternateAssessmentbasedonAlternateAcademicAchievementStandards(AA-AAAS),ispartoftheOregonStatewideAssessmentSystem.TheORExtisadministeredtoOregonstudentswiththemostsignificantcognitivedisabilities(SWSCD)inEnglishlanguageartsandmathematicsinGrades3-8and11.TheORExtisadministeredinscienceinGrades5,8,&11.TheORExtlinkstotheCCSSinEnglishlanguageartsandmathematics.ThenewORExtisduallylinkedtoOregon'sformersciencestandards,aswellastotheNGSS.ResultsfromtheEnglishlanguageartsandmathadministrationsareincludedincalculationsofparticipationandperformanceforAnnualMeasureableObjectives(AMO)–aprovisionoftheNoChildLeftBehindAct(NCLB).ScienceparticipationisalsoincludedaspartoftheTitle1AssessmentSystemrequirements,andisadministeredingrades5,8,&11.TherevisedORExtisbuiltuponaverticalscaleinordertosupportreliabledeterminationsofannualacademicgrowthinELAandmathematicsinGrades3-8.ThecompleteverticalscalingplanandoperationalitemselectiondecisionrulesarelocatedinAppendix2.2.1.

6.1AStateFormallyAdoptedAlternateAcademicAchievementStandards

TheStateBoardofEducationformallyadoptedtheAAASandachievementleveldescriptors(ALDs)onJune25,2015(seeAppendix6.1A.1).TheELA,Math,andScienceAAAS,includingboththeALDsandtherequisitecutscoresareincludedinAppendix6.1.A.2.

6.1BStateAppliesAAAStoAllPublicSchoolSWSCDinTestedGrades

ThestateappliestheAAAStoallpublicschool-servedSWSCDwhoparticipateintheORExtinGrades3-8&11inEnglishlanguageartsandmathematics,andinGrades5,8,&11inscience.

6.1CState'sAAASIncludeAtLeastThreeLevels,ALDs,andCutScoresThealternateacademicachievementstandardsinOregonarecomposedoffourlevels(thoughonlythreearerequired).Indescendingorder,theyare(a)Level1,(b)Level2,(c)Level3,and(d)Level4.Level1andLevel2performancesrepresentproficientachievement,whilethebottomtwolevelsrepresentachievementthatisnotyetproficient.TheproceduresfollowedtodevelopOregon'salternateacademicachievementstandardswereconsistentwithTitle1assessmentsystemrequirements,includingtheestablishmentofcutscores,whererelevant.Inordertodefinefourlevelsofproficiency,Oregonsetthreecutscoresacrossallsubjectareas:(a)toseparateLevel1fromLevel2,(b)toseparateLevel2fromLevel3,and,(c)toseparateLevel3fromLevel4.ThealternateacademicachievementstandardsinEnglishlanguagearts,mathematics,andsciencefortheORExt,includingtheachievementleveldescriptors(ALDs)andcutscores,wereestablishedduringstandardsettingmeetingsheldonJune15(science),16(mathematics),and17(Englishlanguagearts).


130

6.2AchievementStandardSettingStandardSettingmeetingswereheldattheUniversityofOregoninEugene,ORonJune15,2015(Science),June16,2015(Mathematics),andJune17,2015(Englishlanguagearts).Atotalof53standardsetterswereinvolvedintheprocess:11inScience,and21inbothEnglishlanguageartsandMathematics.Panelistswereassembledingradelevelteamsofthree,wheretwomemberswerespecialeducatorsandonememberwasacontentspecialist.Thepanelistswerehighlyeducated.Over90%ofthepanelpossessedaMaster’sdegreeorhigher.Fifty-seven(57%)percentofthepanelistshadover11yearsofteachingexperience.Seventy-sixpercent(76%)ofthepanelistshadsomeexperienceworkingwithstudentswithsignificantcognitivedisabilitieswith64%licensedasSpecialEducators.Themajorityofpanelmemberswerefemale(87%),fromtheNorthwestofthestate(87%),andWhite(83%).Nopanelmemberself-identifiedwithOregon’smajorminoritypopulation(Hispanic).Inadditiontothelivetrainingduringstandardsettingmeetings,panelistswereaskedtocompleteseveraltrainingrequirementspriortothestandardsettingmeetings,whichorientedthemtothestudentpopulationofstudentswithsignificantcognitivedisabilities(SWSCDs),theOregonExtendedAssessmenttestdesignandhistory,aswellasthebookmarkingstandardsettingmethod.Panelistswerequiteconfidentintheirpreparationandfinaljudgments,asevidencedbyresponsestothequestions:(a)"Thetraininghelpedmeunderstandthebookmarkmethodandhowtoperformmyroleasastandardsetter."(b)"Iamconfidentaboutthedefensibilityandappropriatenessofthefinalrecommendedcutscores."and,(c)"Overall,IamconfidentthatthestandardsettingproceduresallowedmetousemyexperienceandexpertisetorecommendcutscoresfortheORExt."Theheartymajorityofstandardsettersstronglyagreedwiththesestatements,whileallparticipantsagreed.Thenine-stepprocessimplementedforthesestandardsettingmeetingswasbasedonHambleton&Pitoniak(2006)asreportedbyR.L.Brennan(EducationalMeasurement,4thEdition,pp.433-470).StandardsettingevaluationquestionsposedtoparticipantswereadaptedfromCizek'sSettingPerformanceStandards(2012).StandardsetterssetcutscoresandrecommendedAchievementLevelDescriptors(ALDs)fortheOregonStateBoardofEducationtoconsider.Thecutscoreswerearticulatedtoreflectverticaldevelopment,oratleastmaintenance,ofexpectationsacrossgradesinamannerthatrespectedstandardsetterjudgmentstothegreatestpossibledegree.SixchangesweremadeinELAandMathematics.Scienceisnotbuiltuponaverticalscale,sonocutscoreadjustmentswerenecessaryinScience.Thecutscoresarelistedbelow.


131

Englishlanguagearts(ELA)

Grade Level1 Level2 Level3 Level43 191orbelow 192-212 213-227 228orabove4 199orbelow 200-212 213-227 228orabove5 201orbelow 202-219 220-231 232orabove6 204orbelow 205-219 220-232 233orabove7 207orbelow 208-221 222-235 236orabove8 212orbelow 213-223 224-235 236orabove11 898orbelow 899-919 920-926 927orabove

Mathematics

Grade Level1 Level2 Level3 Level43 191orbelow 192-200 201-217 218orabove4 192orbelow 193-205 206-218 219orabove5 192orbelow 193-205 206-219 220orabove6 203orbelow 204-207 208-221 222orabove7 206orbelow 207-208 209-222 223orabove8 207orbelow 208-211 212-225 226orabove11 900orbelow 901-906 907-921 922orabove

Science

Grade Level1 Level2 Level3 Level45 505orbelow 506-516 517-529 530orabove8 809orbelow 810-819 820-830 831orabove11 900orbelow 901-913 914-928 929orabove

Note:TheELAandMathverticalscalesfortheORExtarecenteredon200ingrades3-8andcanbeusedtodocumentyear-to-yeargrowth.Noneoftheotherscalesshouldbeusedforlongitudinalcomparisons.AllGrade11scalesareindependentandcenteredon900.Thegrade5Sciencescaleisindependentandcenteredon500,whiletheGrade8Sciencescaleisindependentandcenteredon800.Anindependentauditorevaluatedthebookmarkingstandardsettingprocess.Theauditor'scomprehensivereportcanbefoundinAppendix6.2.2.

6.3ChallengingandAlignedAcademicAchievementStandardsOregoneducatorsinitiallyevaluatednewOregonEssentializedAssessmentFrameworksintworespects.First,educatorswereaskedtodeterminetheappropriatenessofthestandardsselectedforinclusionandexclusionintheEssentializedStandards(yes/no).Second,theleveloflinkagebetweentheEssentializedStandardsandgradelevelcontentstandardwasevaluated(0=nolink,1=sufficientlink,2=stronglink).Summaryresultsareprovidedinthetablesbelow.Acomprehensiveessentializedstandardtogradelevel


132

standardlinkagestudy,aswellasessentializedstandardtoitemalignmentstudy,isprovidedinAppendix3.1A.Englishlanguagearts

Grade#

EssentializedStandards

#Raters Ave.LinkageRating(0-2)*

Ave.AgreementwithEssentialization(0-6)*

3 27(38) 6 1.74(10) 5.68(21)4 30(40) 6 1.78(15) 5.77(25)5 28(39) 6 1.73(12) 5.79(23)6 25(37) 6 1.80(12) 5.76(19)7 24(36) 6 1.77(10) 5.79(19)8 25(36) 6 1.79(12) 5.80(21)11 24(36) 6 1.82(12) 5.79(19)Note.*Countofperfectratings/agreementacrossallraters(inparenthetical)relativetonumberofessentializedstandards.Mathematics

Grade#




3 22(33) 3 2.00(22) 2.77(17)4 26(34) 3 1.99(25) 2.81(21)5 23(34) 3 1.99(22) 2.78(18)6 27(41) 3 1.98(21) 2.68(15)7 20(36) 3 1.95(17) 2.90(18)8 19(33) 3 1.96(17) 2.37(7)11 23(179) 3 2.00(23) 2.52(12)Note.*Countofperfectratings/agreementacrossallraters(inparenthetical)relativetonumberofessentializedstandards.Science

Grade#




5 15(16) 4 1.92(10) 3.93(14)8 24(59) 4 1.97(21) 4.00(24)11 24(71) 4 1.98(22) 3.83(20)Note.*Countofperfectratings/agreementacrossallraters(inparenthetical)relativetonumberofessentializedstandards.


133

6.4ReportingOregon'sreportingsystemfacilitatesappropriate,credible,anddefensibleinterpretationanduseofitsassessmentdata.WithregardtotheORExt,thepurposeistoprovidethestatetechnicallyadequatestudentperformancedatatoascertainproficiencyongradelevelstatecontentstandardsforstudentswithsignificantcognitivedisabilities(seeSections3and4).Inaddition,thestatemakesitclearthatresultsfromtheOregonExtendedarenotcomparabletoresultsfromtheSBA/OAKS(seeAppendix2.3,p.29-31).Nevertheless,thetestmeetsrigorousreliabilityexpectations(seeSection4.1).ValidityisconsideredhereasanoverarchingsummationoftheOregonExtendedassessmentsystem,aswellasthemechanismsthatOregonusestocontinuouslyimprovetheORExtassessment(seeAppendix2.3B.10).

6.4APublicReportingOregonreportsparticipationandassessmentresultsforallstudentsandforeachoftherequiredsubgroupsinitsreportsattheschool,district,andstatelevels.Thestatedoesnotreportsubgroupresultswhentheseresultswouldrevealpersonallyidentifiableinformationaboutanindividualstudent.ThecalculationrulefollowedisthatthenumberofstudentsinthesubgroupmustmeettheminimumcellsizerequirementforeachAMOdecision:participation,achievementinEnglishlanguageartsandmath,attendance,andgraduation,whereappropriate(seeAppendix2.6C)

6.4BStateReportsInterpretableResults

Oregondevelopsanddisseminatesindividualstudentdatauponfinaldeterminationofaccuracy.Thestateprovidesdistrictswithindividualstudentreports(ISRs)thatmeetmostrelevantrequirements.ThestateincorporatedtheStandardErrorofMeasure(SEM)foreachstudentscoreintothereporttemplates.TheSEMassociatedwitheachcutscoreisprovidedinSection4.1B.Also,seethemock-upISRinAppendix6.4C.

6.4C1–C5StateProvidesIndividualStudentReports

Oregon'sstudentreportsprovidevalidandreliableinformationregardingachievementontheassessmentsrelativetotheAAS.ThereliabilityofthedataisaddressedinSection4.1.ValidityisconsideredhereasanoverarchingsummationoftheOregonExtendedassessmentsystem,aswellasthemechanismsthatOregonusestocontinuouslyimprovetheOregonExtendedassessment.TheISRsclearlydemonstratethestudents'scalescorerelativetheAAASthatisrelevantforthatcontentareaandgradelevel(seeSection4.4andAppendix6.4C).TheOregonISRsprovideinformationforparents,teachers,andadministratorstohelpthemunderstandandaddressastudent'sacademicneeds.Thesereportsaredisplayedinasimpleformatthatiseasyforstakeholderstounderstand.Districtrepresentativescantranslateresultsforparentsasnecessary.ScaledscoreinterpretationguidanceispublishedinAppendix2.1A.

2016-2017 OR Extended Assessment Conclusions and Next Steps– Page

134

ConclusionsandNextSteps

Insum,therigoroftheproceduraldevelopmentandstatisticaloutcomesoftheORExtweresubstantiveandsupporttheassessmentsintendedpurpose.Proceduralevidenceincludesessentializedstandardsdevelopment,itemdevelopment,itemcontentandbiasreviews,anindependentalignmentstudyanditemselectionbaseduponitemcharacteristics.Outcome-relatedevidenceincludedmeasurereliabilityanlayses,pointmeasurebiserials,outfitmeansquares,itemdifficultyandpersonabilitydistributions,andconvergentanddivergentvalidityevidence.Thesesourcesofevidencewereallquitegoodandprovideimportantvalidityevidence.ThetestdevelopmentprocessadheredtoproceduralguidelinesdefinedbytheAERA/APA/NCMEStandardsforEducationalandPsychologicalTesting(2014),aswellasincorporatingproceduresthatareknowninthefieldtobebestpractice.Forexample,anindependentauditorevaluatedalignment.Inaddition,theORExtreflectswhathighlyqualifiedOregoneducatorsbelieverepresentsthehighestprofessionalstandardsforthepopulationofstudentswithsignificantcognitivedisabilities,asevidencedinourconsequentialvaliditystudybyteachersupportoftheacademiccontentontheORExtaswellasthebehaviorssampledduringtestadministration.Dr.DiannaCarrizalesconductedanindependentalignmentstudyconsistingoffiveevaluationcomponents:a)standardselectionforessentialization,b)strengthoflinkagebetweenessentializedstandardsandgradelevelcontentstandards,c)alignmentbetweenitemsandessentializedstandards,d)alignmentbetweentheessentializedstandardsandtheachievementleveldescriptors,ande)alignmentbetweentheachievementleveldescriptorsandtheORExttestitems.Dr.Carrizalesreportedthat,“Inthethreeevaluationsthatinvolveddeterminingtherelationshipbetweenstandardsanditems,reviewersidentifiedsufficienttostrongrelationshipsamongassessmentcomponentsinallgradesandallsubjectareas.InthetwoevaluationsinvolvingAchievementLevelDescriptors,reviewersidentifiedthirtyinstancesofsufficienttostrongrelationshipsoutofthirty-fourpossiblerelationshipopportunitiesresultinginanoverallaffirmedrelationshipwithareasforrefinementsidentified.”Overall,documentationcollectedinthereportsuggeststhattheORExtassessmentsystemisaligned.ThetestreliabilitiesfortheORExtwerequitehigh,suggestingthattheassessmentitemsfunctionedconsistentlywiththetestasawhole.Thecorrelationsbetweenstudents’contentscoresacrosssubjectswerenotoverlystrong,implyingthateachtestmeasuresadistinctconstruct.TheclassificationconsistencyanalysesdemonstratethattheORExtisappropriatelycategorizingstudentsintotheproficientcategory,andcapableofdoingsoinaconsistentmanner.Theverticalscaledevelopedin2014-15appearstobemodelingincrementalgrowthacrossGrades3-8inELAandmathematics,asintended.TheGrade7mathematicstestcontinuedtodemonstrateinsufficientitemdifficultiesacrosstherangeoflow,medium,andhighitemcomplexity,however,andmustagainbeamendedinthe2017-18schoolyear.TheELAandscienceassessmentscouldcontinuetobenefitfromtheadditionofmoredifficultitems,asevidencedbycomparisonsoftheaverageperson


135

abilitiesanditemdifficulties.Mathematicsassessmentsappeartobefunctioningquitewellintermsofpersonabilitiesanditemdifficulties,thoughsomeadditionallowlevelitemsmighthelpincreaseaccessforthegroupofstudentsfunctioningatthatlevel.TheOregonObservationalRatingAssessment(ORora)resultsdemonstratethatapproximately17-25%oftheSWSCDwhoparticipatedintheORExtalsotooktheORora,dependingupongradelevel.Atotalof755studentswereadministeredtheORorainthe2016-17testadministration.Theparticipantswereprimarilystudentswithmultiple,severedisabilitieswithverylimitedcommunicationsystems.AnalysesofmissingdatapatternsfortheORExtdemonstratedthatQAsweregenerallyabletoadheretothediscontinuationrules.Quantitativeresultsindicatethatatotalof755studentsacrossalltestedgradeswereadministeredtheORora.ResponsepatternsontheORExtwerecomparedtoORoraresultstodeterminewhatpercentagesofQAswereadministeringtheORoraduetotheminimumparticipationruleandwhatpercentagewereadministeringtheORoraoftheirownvolition.Analysesshowedthat234studentswereeligibletotaketheORorainEnglishlanguagearts,241studentswereeligibletotaketheORorainmathematics,and86wereeligibletotaketheORorainscience.Thismeansthatabout30studentspergrade,percontentareareceivedfiveorfewercorrectresponseswithinthefirst15itemsadministeredontheORExt.Ofthe561testrecordsthatmetORoraeligibilityrequirements,91werenotadministeredtheORora.Inaddition,therewere82studentsinELAandMath,respectively,whowereadministeredtheORorawithouthavingparticipatedintheORExt(74ofthosestudentswerethesamestudents,acrosseachcontentarea,witheightstudentsuniquetoeachcontentarea,respectively).The2016-17OregonConsequentialValiditystudyprovidesimportantinformationforfutureadministrationsoftheORExt.TheresultsdemonstratethatthetestcontinuestobeeasytoadministerandscoreandisprovidinganaccessibleandappropriaterepresentationoftheknowledgeandskillsthatshouldberequiredofSWSCDinOregon.Areasofrequestedimprovementincludetheprovisionofatablet-basedadministration,whichisalreadyplannedfor2017-18,andthedevelopmentofadditionallifeskillsitems,whichcannotbeaccomplishedwhilemaintainingrigorousacademicexpectationsthatarelinkedtoOregoncontentstandards.The2016-17OregonExtendedAssessmentPilotTabletAdministrationdemonstratedthatOregonteachershighlyvalueprovisionofatablet-basedadministrationoftheORExtatthestatewidelevel.Benefitsofatablet-basedadministrationincluded:increasedstudentengagement,improvedstandardization,easeofusebyteachers,andresourceprotection(i.e.,time,printing,expense).Theresultsalsosuggestthatmorerobustsystemsareneededtosupportuseraccesstothetestingapplicationviaanautomaticusernameandpasswordprocess.FocusGroupmembersalsorecommendedthatpracticeitemsbedevelopedinatabletformatsoqualifiedassessorsandstudentscanpracticewiththetabletadministrationinpreparationfortheORExttestwindow.Documentingevidenceofvalidityremainsanongoingandcontinuousprocess.Oureffortstocontinuetoimprovetheassessmentsystemareoutlinedbelow,aswellasinSections3and4above.Wealsohavestudiesplannedoverthecourseofthenextthreeyearsthatwill


136

helptosolidifytheevidencethatisaccumulating.AlloftheevidencewehaveathandsuggeststhattheORExtissufficienttoitsstatedpurposeofprovidingreliabledeterminationsofstudentproficiencyatthetestlevelinordertosupportsystemslevelanalysisofdistrictandstateprograms.TheORExtwillhopefullycontinuetoimproveovertimeduetofield-testingandconstantmonitoringandreview,andadditionalvalidityevidencewillbegathered.AsmentionedaboveinSection3.1A,dataarepresentedtosupporttheclaimthatOregon’sAA-AAASprovidesthestatetechnicallyadequatestudentperformancedatatoascertainproficiencyongradelevelstatecontentstandardsforstudentswithsignificantcognitivedisabilities–whichisitsdefinedpurpose.Inthistechnicalreport,wehaveprovidedcontentvalidityevidencerelatedtotheORExttestdevelopmentprocess(i.e.,essentializationprocess,linkagestudy,distributeditemreview,testblueprint,itemwritertraininganddemographics,anditemreviewertraininganddemographics),ORExttestreliabilityevidence,andORExtconsequentialvalidityevidence.FurtheranalysesoverthecomingyearsareplannedtocontinuethedevelopmentoftechnicaldocumentationforoverallconstructvalidityoftheORExt.Thetechnicaldocumentationplanforthe2016through2019schoolyearsisprovidedbelow:DocumentationDescription

AnticipatedTimeframe

Outcome

ScoringAccuracyStudy Spring2018 Provideinter-raterreliabilitydocumentationfortheORExt.

AccommodationsStudy Spring2019 ProvidedocumentationrelatedtotheimpactofaccommodatedtestadministrationfortheORExt.

ORExtEligibilityStudy Fall2019 ProvidedocumentationoftheconsistencyofIEPteamdecisionmakingwiththeestablishedODEguidelinesforORExteligibility.

ORExtFieldTesting Ongoing Testanditemcharacteristicsarereviewedannually,withoperationalitemsthatarenotfunctioningasintendedreplacedbyfieldtestitemsthatarefunctioningproperly.

2016-2017 OR Extended Assessment Appendix Table and Descriptions– Page

137

AppendixTableandDescriptions

AppendixTableTopic FileName

EssentializedAssessmentFrameworksUserGuide App1.1_EAF_UserGuide

IndependentreviewofrigorofCCSSstandards App1.2_FordhamCCSSReview

MemofromtheOregonGovernorregardingparentopt-outoftestingexpectations

App1.4.1_ExecutiveMemo

ODETestAdministrationManual(TAM)App1.4.2_TAM2016_17

ODEEnglishLearnerProgramGuideApp1.4A.1_ODE_ELProgramGuide

Oregonregulationregardinglanguageofassessment App1.4A.2_OAR581_022_0102

Oregon’sannualreporttothestatelegislatureregardingOregon’spublicschools

App1.5_ODEStateReportCard2016

ORExtTestSpecificationsApp2.1_ORExtTestSpecs

ORExtScaledScoreInterpretationGuideApp2.1A_ORExtScaleScoreInterp

ORExtTestBlueprintApp2.1B_ORExt_TestBlueprint_2017

ORExtItemDevelopmentProcessManualApp2.1C_ORExt_ItemDevt_Process

BRTItemWriterTrainingPPTSlidesApp2.2.1_ItemWriter_Training

ORExtVerticalScalingPlanApp2.2.2_ORExtVertScale

ORExtSampleItemsApp2.2.3_ORExtSampleItems

ORExtAdministrationManualApp2.3_ExAssessAdminMan2016_17

ODEAccessibilityManual App2.3A.1_ODEAccessibilityMan2016_17ODEAccommodationsGuide App2.3A.2_OSAAccomGuideBRTProceduresforReducingDepth,Breadth,andComplexityofItems App2.3A.3_ORExtRDBC

ODEQualifiedTrainerPPTSlides App2.3B.1_QATraining2016_17BRTQualifiedTrainerPPTSlides App2.3B.2_ORExtendQTTrng2016_17


138

Topic FileNameODEStatewideAssessmentTestSchedule App2.3B.4_TestSchd_2016_17QualifiedTrainerSuggestedAgenda App2.3B.5_QT_Training_Agenda2016_17HowtoAccesstheor.k12test.comWebsite App2.3B.6_ExtAssessAccessInstr2016_17QualifiedTrainerResponsibilities App2.3B.7_TrainerResponsibilities2016_17FrequentlyAskedQuestionsDocument App2.3B.8_ORExtFAQ2016_17HelpDeskTechnicalAssistanceLog App2.3B.9_HelpDeskLog2016_17ConsequentialValidityStudyReport

App2.3B.10_ORExtCVStudy2016_17

ORExtPilotTabletStudyReport–Phase2App2.3C_ORExtPilotTabletStudyReport

DataEntryGuideApp2.6B_G3-8_11_DataEntryGuide2016_17

ReportCardRatingPolicyTechnicalManual App2.6C_ReportCardRating_PolicyTechManual

IndependentORExtItemLinkage,Bias,andSensitivityStudyReport App3.1A_LinksAlignmentORExtSpring2017

OverviewoftheDistributedItemReviewwebsiteusedforconductingalignment,bias,andsensitivityreviews

App3.1B_DIR_SystemIntro

HistoryoftheORExtDocumentApp4.1B_HistoryORExt_2017

ReportconveyingtheethnicdiversityofOregon’sstudentsforthe2014-15schoolyear

App4.2_FallMembershipReport2016_17

ODEEligibilityGuidancefortheORExtApp5.1B_ORExtEligibilityGuidance

Resultsfromthe2016-17administrationoftheORoraandadministrationandscoringinstructions.

App5.1D_OroraResults2016_17

AgendaandminutesfromtheStateBoardadoptionoftheAAASfortheORExt App6.1A.1StateBoardAAASAdoption

AllAAASfortheORExt,includingALDsandcutscores App6.1A.2_ORExtAAAS

StandardSettingTrainingPPTSlidesApp6.2.1_ORExt_Standard_Setting

IndependentStandardSettingAuditReportApp6.2.2_ORExtSSAuditRept

ORExtIndividualStudentReportMock-UpApp6.4C_ORExt_ISR


139

AppendixDescriptions

Appendix1.1Appendix1.1explainsthedevelopmentprocessandintendedusesfortheEssentializedAssessmentFrameworks(EAFs).TheEAFsaretheessentializedstandards(EsSt),whicharelinkedtogradelevelcontentstandards.TheORExtisalignedtotheEAFs,aswell.WhiletheEAFsprimarilyguideitemdevelopment,theyarealsointendedtobeusedinthedevelopmentofappropriatePresentLevelsofFunctionalandAcademicPerformance(PLAAFP)statementsandIndividualizedEducationProgram(IEP)goalsandobjectives.

Appendix1.2

Appendix1.2conveystheevaluationconductedbyresearchersattheFordhamInstitute,whichcomparedthen-currentstatestandardstotheCCSSintermsofrigor.ThefindingsgenerallyshowthattheCCSSareasrigorousormorerigorousthanstatestandards.

Appendix1.4.1Appendix1.4.1istheExecutiveMemofromtheGovernorofOregonregardingparentopt-outexpectations.

Appendix1.4.2Appendix1.4.2isthetestadministrationmanual(TAM)forallassessmentsintheOregonstatewideassessmentsystem,includingtheSBA,OAKS,theORExt,theKindergartenAssessment,andtheELPA.TheTAMelaboratesallrelevanttestsecurityandadministrationprocedures.

Appendix1.4A.1Appendix1.4A.1isODE'sEnglishLearnerProgramGuide,outliningEnglishlearner(EL)systemrequirementsintheareasofstudentidentification,services,reporting,andassessmentforELsinOregon'spublicschools,includingELswhoareSWD.

Appendix1.4A.2

Appendix1.4A.2isOregon’sregulationsthatrequireODEtoprovidetranslatedOAKSassessmentsforpopulationsatorabove9%ingradesK-12withinthreeyearsaftertheschoolyearinwhichthelanguageexceedsthethreshold.

Appendix1.5Appendix1.5isOregon’sannualreporttothestatelegislatureforthe2015-16schoolyear.Thereportincludesstudentdemographicsandinformationonstudentgroups,schoolfundingandstaffinformation,testresults,graduationanddropoutrates,charterschooldataandinformationonalternativeeducationprograms,earlychildhooddata,andattendanceandchronicabsenteeismdata.


140

Appendix2.1Appendix2.1isthetestspecificationsdocumentthatdescribesourapproachtoassessmentandtestdesignfortheORExt.ThedocumentincludesourapproachtoRDBC,anoverviewoftheessentializationprocessandEAFdocuments,theanticipatedoperationaltestdesignfortheORExt,testdevelopmentconsiderations,sampletestitems,itemspecifications,anduniversaltools/designatedsupports/accommodations.

Appendix2.1AAppendix2.1AprovidesthefieldwithcomprehensiveinformationrelatedtoscaledscoreinterpretationfortheORExt.Theguidanceispublishedinthreemainareas:1)Annualperformance,2)Annualgrowth,and3)Performanceforverylowfunctioningstudents.Guidanceregardinguseandinterpretationofreadingandwritingsubscoresisalsoprovided.

Appendix2.1BAppendix2.1BisthetestblueprintfortheORExt,conveyingthebalanceofrepresentationofdomainsacrossthecontentareasandgradelevelsassessed.Operationalitemsareselectedtoreflecttherepresentationpercentagesincludedinthetestblueprint.

Appendix2.1CAppendix2.1Cdescribestheeight-stepitemdevelopmentprocessusedtodevelopitemsfortheORExt,fromstandardselectiontotestbookletformation.Theitemdevelopmentprocessisspecificandexplicitinordertoincreasetransparency.

Appendix2.2.1Appendix2.2.1isthesetofPPTslidesthatwereusedtotrainitemwritersfortheORExt.Itemwriterswerealsoprovidedanorientationtothetestspecificationsaspartoftraining.

Appendix2.2.2Appendix2.2.2isadocumentthatsummarizesthebalanceddesignverticalscalingplanemployedfortheORExtinthe2014-15administration.Thedocumentincludesthedomainsamplingplanforallassessments,aswellasthedecisionrulesemployedtoremoveitemsfromtheoperationalitempoolpriortoverticalscalingandstandardsettingprocedures.

Appendix2.2.3Appendix2.2.3providesstakeholderswithvisualrepresentationofthestructureoftheORExt.SampleitemsareconveyedinEnglishlanguagearts,mathematics,andscience,withthescoringprotocolandstudentmaterialspresentedtogether.Stakeholderscanseethestructureofeachitem,aswellashowtheitemsarescored.Theycanalsogatheranideaaboutthetypesofformatsthatareusedforanswerchoicesthatareincludedwithinthestudentmaterialsdocuments.


141

Appendix2.3Appendix2.3isODE'sGeneralAdministrationandScoringManualfor2016-17.ThemanualestablishesODE'sexpectationsregardingthetestwindow,utilizingtheORExttrainingandproficiencywebsite,usingthesignlanguageinterpretertrainingandproficiencywebsite,andinformingparents.Italsoprovidesthefollowinginformationforstakeholders,includingeducatorsandparents:

• OverviewoftheExtendedAssessments• AssessingaStudent• Scoring• DecisionMaking• InformationforTeachers

Themanualprovidesthreeappendicesthatprovideguidanceregardingtheprovisionofsupports,parentquestionsandanswers,andaglossary.

Appendix2.3A.1Appendix2.3A.1isthe2016-17accessibilityoptionsmanualforallassessmentsintheOregonstatewideassessmentsystem,includingtheSBA,OAKS,theORExt,andtheELPA.OptionsincludeUniversalTools,DesignatedSupports,andAccommodations.Themanualprovidesguidanceregardinguseoftheseoptionsininstructionandassessment,aswellasimplementationstrategiesanduseevaluation.EachaccommodationiscodedforuseindataanalysisrelatedtoassessmentscoresfortheSBAandOAKS.

Appendix2.3A.2Appendix2.3A.2isODE'sHowtoSelect,Administer,andEvaluateAccommodationsonOregon'sStatewideAssessmentmanualfor2013-14.Themanualtrainsusersregardinghowtoimplementandevaluateappropriateaccommodations,fromthestudentleveltothesystemslevel.

Appendix2.3A.3Appendix2.3A.3isadocumentthatsummarizestheproceduresusedduringitemdevelopmenttoreduceitemdepth,breadth,andcomplexity,inadditiontothetestspecificationsinformationfoundinAppendix2.1.Thedocumentalsoprovidesmoredetailregardinghowlanguagecomplexityisaddressedandreviewedinanefforttodecreasethelanguageloadofitemsandmakethetestmoreaccessibletoallstudents.Thedocumentalsodiscusseswaysinwhichbiasisaddressedduringtestdevelopment.


142

Appendices2.3B.1-2.3B.2Appendices2.3B.1and2.3B.2arethePowerPoint(PPT)trainingsthatwereusedbyODEandBRTtrainerstotrainnewqualifiedassessors(QAs)andqualifiedtrainers(QTs)infourregionallyhostedtrainingsinNovember2016.QTsalsousedthepackagetotrainNewQualifiedAssessorsforthe2016-17schoolyear.ThetrainingprovidesparticipantswiththeinformationneededtopassproficiencytestsaspartoftherequirementstobecomeaQAfortheOregonExtendedAssessmentsandwasdeliveredbyQTsthroughoutthestate.Thetrainingpackageaddressesthefollowingtopics:

• What'snewin2016-17• 2017TestWindow• Eligibility–whichstudentstakeAA-AAAS?• Testadministration• StudentConfidentiality&TestSecurity• TestAdministration(Physical&Logistic)• Scoring&DataEntry• Reports&SharingResultswithParents• NavigatingtheTrainingandProficiencywebsite• Resources

Appendix2.3B.4

Appendix2.3B.4isthetestcalendarfortheentireOregonstatewideassessmentprogram,includingtheSBA,OAKS,theORExt,theELPA,theKindergartenAssessment,andtheNAEP.

Appendix2.3B.5Appendix2.3B.5isasampleagendathatODEmakesavailabletoQTsaroundthestatetotraintheirrespectivenewQAsastheyimplementthetrain-the-trainersmodelusedbytheOregonExtendedassessment.

Appendix2.3B.6Appendix2.3B.6isthelistofinstructionsprovidedtonewQAsandQTsregardinghowtoaccesstheonlinetrainingandproficiencywebsite.

Appendix2.3B.7Appendix2.3B.7isthelistofresponsibilitiesassociatedwithbeingaQTfortheORExtassessment.

Appendix2.3B.8Appendix2.3B.8isthedocumentthatcontainsthemostcommonlyfieldedquestionsandanswersfromstakeholders,includingparentsandteachers.

Appendix2.3B.9Appendix2.3B.9isthereportthatsummarizesallofthetechnicalassistancequestionsgarneredfromthefieldthisyear.Effortsaremadetofindanypatternsthatourteammayusetoimprovetrainingforthefollowingyear.


143

Appendix2.3B.10Appendix2.3B.10istheconsequentialvalidityreportforthespring2017consequentialvaliditystudyconductedbyBRT.ThereportprovidesdocumentationoftheperceptionsinthefieldrelatedtobothintendedandunintendedacademicandsocialconsequencesoftheORExt.

Appendix2.3CAppendix2.3CistheORExtPilotTabletAdministrationreportforthespring2017tabletadministration,Phase2,studyconductedbyBRT.Thereportprovidestheresearchplan,summariesofresults,andlessonslearnedregardinghowtoapproachstatewideoperationaltabletadministrationplannedfornextyear.

Appendices2.6Appendix2.6istheguidancethatODEhasprovidedtoassessorstowalkthemthroughtheonlinedataentryprocessfortheORExtonthesecureODEDistrictSecurewebsite.

Appendix2.6CAppendix2.6CisthemanualdefiningthestateofOregon'spoliciesandproceduresregardinghowstudentsareincludedinAMOreporting,includinghowachievement,growth,andgraduationratesarereportedforstudentgroupsandsubgroups.

Appendix3.1AAppendix3.1AisadocumentthatsummarizestheindependentalignmentstudyprocessandparticipantsusedtoreviewthelinkagebetweentheEssentializedStandardsandgradelevelcontentstandards(CCSSinELAandMath;ORSciandNGSSinScience),aswellasthealignmentbetweentestitemsfortheORExtwiththoseEssentializedStandards.Inaddition,reviewersratedtheitemsforpotentialbiasandaccessconcerns.AlldatawasgatheredusingtheDistributedItemReview(DIR)website,supportedbyawebinartrainingandongoingtechnicalassistance.Theresultsofthe2014-15LinkageStudy,whichwasnotindependentbutrunbyBRTresearchers,arealsoincluded.

Appendix3.1BAppendix3.1BisadocumentthatdescribestheDistributedItemReview(DIR)websiteusedbyOregonteacherstoevaluatethealignmentbetweentestitemsfortheORExtwithEssentializedStandards.Inaddition,reviewersratedtheitemsforpotentialbiasandaccessconcerns.AlldatawasgatheredusingtheDIRwebsite,supportedbyawebinartrainingandongoingtechnicalassistance.

Appendix4.1BAppendix4.1BconveysthehistoricaldevelopmentoftheORExtfrom1999tothepresent,includingthegradelevels/bandsassessed,contentareasassessed,andthetargetedcontentstandards.


144

Appendix4.2Appendix4.2includesthemostcurrentpublishedstateleveldataregardingOregon’sethnicdiversity.

Appendix5.1BAppendix5.1BistherevisedandrigorousguidancethatODEhasprovidedtoIEPteamstoassisttheminmakingappropriateassessmenteligibilitydeterminationsforstudentswithdisabilities.

Appendix5.1DAppendix5.1DincludesasummaryreportofthestatewideresultsandtheadministrationandscoringinstructionsforthenewOregonObservationalRatingAssessment(ORora).TheORoraisadministeredtoallstudentswhoseORExttestingwasdiscontinued.ItprovidesinformationregardingstudentprogressintermsoffunctionalskillsinadaptiveandcommunicationdomainsforthesmallsubgroupofstudentswhoareunabletomeettheacademicexpectationsintheORExt.

Appendix6.1A.1Appendix6.1A.1istheagendaandminutesthatdocumentthehearingandadoptionoftheAAASfortheORExtonJune25,2015.

Appendix6.1A.2

Appendix6.1A.2includesalloftheachievementleveldescriptors(ALDs)andcutscoresthatdefineperformancefortheORExtinqualitativeandquantitativefashions,respectively.TheseAlternateAcademicAchievementStandards(AAAS)describewhatstudentsshouldknowandbeabletodobasedupontheirperformanceontheORExt.

Appendix6.2.1

Appendix6.2.1isthePPTslidesusedtotrainstandardsettersduringtheJune2015standardsettingmeetingsforELA,math,andscience.

Appendix6.2.2Appendix6.2.2isastandardsettingreportgeneratedbyanindependentauditor.ThereportprovidesacomprehensiveevaluationofthebookmarkstandardsettingprocedureemployedfortheORExtonJune15-17,2015.

Appendix6.4CAppendix6.4Cisadocumentthatdisplaystheindividualstudentreport(ISR)thatODEpublishesforstudentswhoparticipateintheORExt.Themock-upincludescutscoresandachievementleveldescriptors(ALDs),aswellaslinkstotheODEwebsiteforadditionalinformation.

Documents

Oregon Department of Education 2016–2017 Technical Report€¦ · Oregon’s Alternate Assessment System Technical Report: Peer Review Documentation: Critical Elements 1-6 It is