Embed Size (px)
Citation preview
Planning for Transition: Evolution of Next-Generation Assessment System Designs to Support Ongoing Improvements in Measuring Complex Skills and ConstructsSkills and Constructs
NCSA 2013NCSA 2013National Harbor, Maryland
1
US Dept. of Ed has funded two consortia of states ith d l t t f twith development grants for new assessments
aligned to Common Core State Standards• Rigorous assessment of progress toward “college and
career readiness”• Common cut scores across all Consortium states• Common cut scores across all Consortium states• Provide both achievement and growth information • Valid, reliable, and fair for all students, except those with , , , p
“significant cognitive disabilities”• Administered online
U lti l• Use multiple measures• Operational in 2014-15 school year(F d l R i t / V l 75 N 68 / F id A il 9 2010 18171 85)(Federal Register / Vol. 75, No. 68 / Friday, April 9, 2010 pp. 18171-85)
2
Member States
3
A Balanced Assessment SystemSummative:
College and career readiness
Common Core State Standards
f
Common Core State Standards
f
All students leave
All students leaveTeachers and
assessments for accountability
specify K-12
expectations for college
specify K-12
expectations for college
leave high school
college and career
d
leave high school
college and career
d
Teachers and schools have
information and tools they need
t ifor college and career readiness
for college and career readiness
ready ready to improve teaching and
learning
Interim:Flexible and open
assessments, used for actionable feedback
Formative resources:Supporting classroom-
based resources to improve instruction
4
Scale and Scope• Population
– Over 19 million students;– Tests in grades 3-8 and grade 11Tests in grades 3 8 and grade 11– All students participate except the severely cognitively
impaired• C• Content
– Assess the breadth of Common Core State Standards– Tests in Mathematics and English Language ArtsTests in Mathematics and English Language Arts– Support summative and interim systems– Requires over 20,000 items and tasks
Page 5
Conditions for item innovation
• Presentation options• Response optionsTechnical Toolbox Response options
• Adaptive and Performance Task deliveryDelivery & Scoring
• Emphasis on understanding and i ti
• Scoring methods
C t t
Delivery & Scoring
communication• Integration and application of skills
C t ti f i d
Content
• Current conceptions of mind • Effects on theories of thinking and learningCognitive orientation
Page 6
Cognitive psychologyInfluence on assessmentInfluence on assessment
• Task development • Seek to reveal information processing or mental operations of the examinee.U d l th f iti i d• Use a model or theory of cognitive processing and internalized knowledge structure.
• Examination of how students develop problemExamination of how students develop problem solving techniques and internal representations of subject matter
• Require an explicit path stating why a specific response to a task shows what examinees do mentally or how they represent knowledgementally or how they represent knowledge.
Page 7
Evidence-Centered Design
Observation Interpretation
“A t“AssessmentTriangle”
Cognition
• Item development approach
• Evidence needed to support
• Types of items and tasks needed toapproach
that defines claims about st dents and
to support claims
needed to collect evidence
students and their learning
8
Content - What’s on the test?Content What s on the test?
• Common Core State Standards- Alignment with expectations for college & career success;- Clarity;
Consistency across all states;- Consistency across all states;- Content integration, application through high-order skills;- More coherent than current state and international
standards;- Reality-based, for effective use in the classroom
Evidence and research based- Evidence and research-based http://www.corestandards.org/resources/frequently-asked-questions
9
Approved Claims: MathContent – Approved Math Claims
• Overall Claim– Concepts and Proceduresp– Problem Solving– Communicating Reasoning
M d li d D t A l iReported together– Modeling and Data Analysis
Content – Approved ELA Claims
together
• Overall Claim– Reading
pp
g– Writing– Speaking and Listening
R h/I i
Page 10
– Research/Inquiry
Item and Task Specifications• Begin with CCSS• Define claim• Define assessment target• Define required evidenceq• Develop task model• Develop item/task prototypesDevelop item/task prototypes
Content Standard Claim
Assessment Target
Evidence Item/Task
Page 11
Delivery & ScoringDelivery & Scoring• Test blueprints
• Each student has an adaptive and a PT section• Scoring of CAT items is in real time: CAT items are machine scored or AI scoredmachine scored or AI scored
• PT scoring is part real time, part human scored• Overall and scale scores are calculated from CAT and PT responses
• The adaptive nature of the test means that every category required by the blueprint has toevery category required by the blueprint has to be available at every difficulty level—includes DOK
12
Item ScoringItem Scoring• Machine scoring
A t t d i i l ti• Automated scoring in real time• Includes MC, drag/drop, sorting, matching• Fast, accurate AI i• AI scoring• Requires training an AI engine with human scored responses • may not be available in real time• The most accurate engines are purely statistical. • Fast (after training), not so accurate
• Human scoringg• Scored by educators under controlled conditions• Most favored by higher ed and content experts• Expensive, not in real time
13
p ,
Technical Toolbox• Smarter started out with these item types:
Technical ToolboxSmarter started out with these item types:– Selected Response– Constructed Responsep
– Technology enabledgy– Technology enhanced– Performance tasksWe learned a lot in pilot development
Page 14
Tech Toolbox:Tasks may have any combinations of these y y
featuresPresentation Type Response Type Scoring Type
Static text Select from list Automatic
Graphics Manipulate objects Automatic with rubric tweaks
Video Enter text AI engine‐essay
A di N i AI i h tAudio Numeric response AI engine‐short answer
Animation Construct Equation AI scoring‐equation
Simulation Sequence, order Hand scored
Highlight
15
. . . Highlight . . .
ELA example-highlight text
16
ELA example - Sort/order
17
Item 43083: The Contest
Part C: How many four‐eyed space creatures are needed to make a group with 24 y y p g ptotal eyes? (grade 3)
Part D: Somebody told the five‐eyed space creatures that they could not join the contest Explain why five eyed space creatures cannot make a group with 24 total
Slide 18
contest. Explain why five‐eyed space creatures cannot make a group with 24 total eyes.
Reusable design elementGrid with points and linesGrid with points and lines
Manipulate objects
Slide 20
Drag & Drop
Slide 21
Water Tank - simulationTwo water tanks are shown. Tank A is a rectangular prism and Tank B is a li d Th t k t d tcylinder. The tanks are not drawn to
scale.
Tank A is filled with water to the 10‐meter mark.
Click Tank A to change the water level. The volume of water that leaves TankThe volume of water that leaves Tank A is transferred to Tank B, and the height of the water in Tank B is shown.
Drag one number into the box to show the approximate radius of the base of Tank B.
Slide 22
Equation Editor
23
Lessons from pilot
• As soon as possible, review items as the student will experience them, with same resolution, controls etccontrols, etc.
• Make sure technical folks are involved throughout. – Avoid fantasy overreach– Take advantage of tech knowledge and solutions
• Make sure item spec designers are aware of technical possibilities, limitations p
• HEADSETS! Get them early and have lots available
• Don’t hesitate to innovate “just because you can”Don t hesitate to innovate just because you can
Page 24
All Smarter Balanced members will receive core services and access to additional optional services
‘Core package’ bundle that all members receive as base‐ level services • Summative assessments
R l d d if bili d li bili d• Rules and procedures to certify comparability and ensure reliability and scoring consistency
• Reporting and analytics research, validity studies, peer review documentationdocumentation
• Membership services and communications • Additional technical support for state accountability
Incremental services2, available “for purchase” on top of core package
digital library of i t i t
Item bank and test admin for ddi i l i
g yformative tools and
resources
interim assessment system
additional secure, summative assessment(s)1 (e.g., end‐of‐course or 12th grade)
1 2 3
25
1 Combines security of summative assessment and flexibility of the interim (on test blueprint flexibility)2 Costs for incremental services will have a Smarter component for the oversight portion, and a vendor component for the administration3 Additional technical support service‐hours beyond basic model to be negotiated as incremental fee‐for‐service on state‐by‐state basis
Smarter’s Role in New Item Development
• Consortium gets items from a combination of vendor and state procurement
• Opportunities for states to participate in itemOpportunities for states to participate in item development and review
• About a third of the sustaining budget is expected to be d t d t it d t t d l t i l didevoted to item and content development, including development of new item types
• Smarter will establish rules, standards and procedures to certify item development bodies
• Will develop procedures for “incubating” new item types
26
