Embed Size (px)
Citation preview
1
Accelerating Future Possibilities for
Assessment and Learning
Eva L.BakerUCLA Graduate School of Education & Information Studies
National Center for Research on Evaluation,Standards, and Student Testing (CRESST)
2005 CRESST Conference
September 9, 2005 – Los Angeles, CA
The findings and opinions expressed in this presentation do not reflect the positions or policies of the Institute of Education Sciences or the U.S. Department of Education
2
Premise
The rest of society has made trade-offs not yet confronted by the assessment R&D community
NOW rather than LATER PARTIAL rather than FULL solutions LOWER COST rather than HIGHER QUALITY POLICY CLAIMS accepted PRESSURE cycle for external change TRUTH and stability have other sources than
empirical evidence Institutional memory aging fast
3
Need to Address
or abandon key tenets, validity or lose resources in the future
4
Outcomes of R&D Processes
Quality – Maintenance improvements take time and
have costs Speed
– Faster at same level of quality means cost – Speed required by policy with less or equal
cost Apply to development of interventions
and assessments; evaluation and research activities
5
Quality—Improvements Take Time and Have Costs
http://www.bentleyboys.com/images/sr%20pictures/UFF341front.JPG
6Speed and Quality—Faster at Level Quality Means Cost
http://www.automotriz.net/2003/galeria/images/1024x768-Porsche-911-Carrera-4-01.jpg
7
Give Up to Get:A Standard Tradeoff
http://www.pricequotes.com/cars/submissions.phtml?form=pq&make=560&model=160:2005&source=gl
8
Change the Terms
http://homepage.tinet.ie/~watty/tcz/picz/tng/tranroom.jpg
9
Speed, Quality, Reduced Cost
http://www.gamespot.com/pc/action/startrekvoyagereliteforce/screens.html?page=89
10
In the Meantime, Expect Conventional Customer Behaviors
Push policy requirements (e.g., time boundaries), with occasional successes
Buy on the basis of availability and claims to meet mandates
Resist understanding processes or consequences
Recognize that remedies take time and money that are not available
11
Inferences for Assessment Community
Practitioner demand for quality assessment R&D is dropping
Quality (e.g., validity evidence) not high on list
Five-year time frames are rare and rarer
Time to change or lose the game
12
New Goals for Assessment R&D with Speed and Lower Cost
Show benefit over current option(s)– Value added
Simultaneously maintain or reduce costs
Increase velocity of outcomes (research, development, evaluation)
Increase appropriate interactivity with market and individual consumers
13
Take Assessment
Increase quality: Technical merit goals – Improve fidelity of measures– Improve evidence for multiple purposes– Demonstrate use of new assessments
results in better outcomes State measures and transfer beyond themSpeed of learning curve
– Impact standards—the two-way street
14
Strategy 1: Tried and True Surfaces
http://www.learningtoflyoriginals.com/Miscellaneous/slides/oreos1a_final.html
Give people what they think they want
Surface familiarity Switch the insides Demonstrate benefit
15
Marginal Surface Changes
http://images.google.com/imgres?imgurl=http://www.westons.com/acatalog/kle538t_large.jpg&imgrefrl=http://www.westons.com/acatalog/info_KLE538T.html&h=600&w=800&sz=53&tbnid=GE1pdnfIG90J:&tbnh=106&tbnw=142&hl=en&start=79&prev=/images%3Fq%3Dkleenex%2B%2B%26start%3D60%26svnum%3D10%26hl%3Den%26lr%3D%26ie%3DUTF-8%26sa%3DN
16
Strategy 2: Light Show, Bells, Whistles, and Value
http://www.learningtoflyoriginals.com/Miscellaneous/slides/fireworks.html
Glitzy new thing Gotta have Quality and low cost
17The CRESST Assessment Planning Team
http://www.etsu.edu/physics/etsuobs/starprty/22099dgl/planalign.htmhttp://movies.yahoo.com/shop?d=hc&id=1804851147&cf=pg&photoid=521818&intl=us
18
Four Ways to Use Learning in Assessment Design
1. Big Ideas– Principles– Themes– Schema
Assessment benefit Reusable, recurrent, spiral
Procedure Retrofit to Standards
19
Learning in Assessment Design (Cont’d)
2. Knowledge representation– Show relationships within and among key
content and tasks– Requires designer to have coherent view
Assessment benefit Map to monitor progress Develops teacher knowledge
Procedure Template based for design and administration,
automated scorings
20
NEWTON'SLAWS
Third Law
Second Law
First LawA body in motionremains in motion
unless...
Forces betweeninteracting bodies:equal but opposite
is
is
is
areForce equals Masstimes Acceleration
(F=MA)
Serious Knowledge Representation
21
V.2, 9.7.05
21
22
Learning in Assessment Design (Cont’d)
3. Explanation– Substantial evidence for instructional and
self-assessment
Assessment benefit Invokes informational feedback Diagnostic value Coherence
Procedure Use existing scoring rubrics, peer assessment, or
automated scoring, or selection of best explanations
23
Some Explanations Are Hard
Like Hempel, van Fraassen seeks to explicate explanation as a purely logical concept. However, the logical relation is not that of premises to conclusion, but one of question to answer. Following Bromberger, van Fraassen characterizes explanation as an answer to a why-question. Why-questions, for him, are essentially contrastive. That is, they always,implicitly or explicitly, ask: Why Pk, rather than some set of alternatives X=? Why-questions also implicitly stipulate a relevance relation R, which is the explanatory relation (e.g., causation) any answer must bear to the ordered pair.
http://www.utm.edu/research/iep/e/explanat.htm
24
Other Explanations Are Clear
Why were Sally and I bored?“Too wet to go out and too cold to play ball
So we sat in the house and did nothing at all”
http://trains4tots.com/catinhatbods.html
25
Learning in Assessment Design (Cont’d)
4. Integrated worked examples– Provide total picture of goal in procedural or
problem solving (Sweller, Mayer, et al.)
Assessment benefit Complexity scaled a priori and empirically Integration with instruction Working memory limits
Procedure Templates
26
A Worked Example
1) c(a+b) =f 1a) g(a+m) =k (a+b) =f/c a =f/c-b
2) ac+e =g 2a) ab+f =h ac =g-e a =g-e c
http://education.arts.unsw.edu.au/CLT_NET_Aug_97.HTML
27
All Four Strategies Support Learning, Teaching, and
Transfer New CRESST award integrates all 4
– In a formative assessment plan– Experimental treatments in States– In assessments of pre-Algebra–Algebra
over the course of 3 years– Impact on learning– Assessing transfer and relative speed of
acquisition
28
PRINCIPLESSCHEMA DOMAIN REPRESENTATIONEXPLANATIONWORKED EXAMPLES PROVIDED ASSESSMENTS & TEMPLATES
FOR AMERICA’S CHILDRENPOWERED BY CRESST
29
For Other Tasks atOther Levels
30
Using Cognitive Demands:
Problem Solving
Teamwork and Communication
31
Collaboration and Meta-Cognition
32
Quality, Faster, Cheaper
Implementation– Early buy-in to
administrative hassle– Web support– Community of
practice– Interactive help– Linked to instruction
and emerging teacher expertise
Assessment– Reusable templates– Validity up front in
development– Domain maps made
available – Assessments free– Ambitious goals
33
Help Us Show That Assessment Research Can
Work Smarter, Better,at Lower Cost
