Upload
christal-beasley
View
214
Download
0
Tags:
Embed Size (px)
Citation preview
C R E S S T / U C L A
Eva L. Baker
International Congress for School Effectiveness and ImprovementJanuary 2003
Sydney, Australia
UCLA Graduate School of Education & Information StudiesCenter for the Study of Evaluation
National Center for Research on Evaluation, Standards, and Student Testing
Technology-Based Assessment for
High-Performance Learning
C R E S S T / U C L A
Premises and Goals
Assessment is central to the effectiveness of classroom and distance learning and accountability—some basics
What are advances in assessment? How does technology fit?
How can technology weave instructional and external testing into a coherent system?
Can assessment be cost-sensitive and valid?
Many examples throughout
C R E S S T / U C L A
Order of Topics
Basics about assessment
CRESST research-based models
Attributes, benefits
Examples
Template 1: Paper and pencil
Template 2: Computer
Template 3: Authoring
Technology benefits and requirements
C R E S S T / U C L A
About Assessment Role of external tests—accountability, evaluation,
and system monitoring
Instructional uses of tests in classrooms and schools: diagnosis, modeling, formative assessment as a core teaching strategy
Variations in curriculum require strands supporting coherence among goals (standards), content, cognitive demands (what thinking skills are required?)
Coherence perceived from student, teacher, administrator, and expert views
C R E S S T / U C L A
External Instructional Exams Assessments
Motivated performance
Time-sensitive
Standardized
Shallow sampling
Stand-alone
Embedded in learning
Adapted to learners
Extended time
Opportunity to revise
Contextualized results
C R E S S T / U C L A
External and Instructional Assessments Must Be More
Coherent
Horizontally and vertically
Conceptual and psychological linkage
For accountability systems, classroom measures may be good supplements
C R E S S T / U C L A
CRESST Assessment Models
Do not start with content
Focus on aspects of learning and assessment that transfer from content to content area
Multi purpose: both formative learning and outcomes
Emphasize student-constructed answers
Expert performance defines scoring
Research substantiates these models across different subjects and people
C R E S S T / U C L A
Families of Cognitive Demands:
Model-Based Assessment
ContentUnderstanding
ProblemSolving
Teamwork andCollaboration
MetacognitionCommunication
Learning
C R E S S T / U C L A
CRESST Assessment Models
Research-based, and sites are exclusively classrooms, schools, and systems
Focus on cognition and learning
Combine content-independent and content-dependent knowledge and strategies
Reusable and cost-sensitive
C R E S S T / U C L A
Examples in Knowledge Understanding and Problem
Solving
Paper-pencil templates
Technology-based administration, scoring, and reporting
Technology-supported authoring templates and menus
For teachers, curriculum experts, test makers
C R E S S T / U C L A
Assessment of Understanding
Deep understanding of primary source materials or key processes
Standard reading as part of task
Standard directions
Standard scoring rubrics based on experts’ performance
C R E S S T / U C L A
Hawaiian History Assessment Task:
Bayonet Constitution
Be sure to show the relationships among your ideas and facts.
Your essay should be based on two major sources:
1. The general concepts and specific facts you know about Hawaiian history, and especially what you know about the period of the Bayonet Constitution.
2. What you have learned from the readings yesterday.
Imagine you are in a class that has been studying Hawaiian history. One ofyour friends, who is a new student in the class, has missed all the classes.Recently, your class began studying the Bayonet Constitution. Your friend isvery interested in this topic and asks you to explain everything that you havelearned about it.
Write an essay explaining the most important ideas you want your friend tounderstand. Include what you have already learned in class about Hawaiianhistory, and what you have learned from the texts you have just read. Whileyou write, think about what Thurston and Liliuokalani said about the BayonetConstitution, and what is shown in the other materials.
C R E S S T / U C L A
Excerpts from Hawaiian HistoryPrimary Source Documents
LILIUOKALANI For many years our sovereigns had welcomed the advice of American residents who had established industries on the Islands. As they becamewealthy, their greed and their love of power increased. Although settledamong us, and drawing their wealth from resources, they were alien to usin their customs and ideas, and desired above all things to secure their own personal benefit.
Kalakaua valued the commercial and industrial prosperity of his kingdomhighly. He sought honestly to secure it for every class of people, alien ornative. Kalakaua’s highest desire was to be a true sovereign, the chiefservant of a happy, prosperous, and progressive people.
And now, without any provocation on the part of the king, having maturedtheir plans in secret, the men of foreign birth rose one day en masse, calleda public meeting, and forced the king to sign a constitution of their ownpreparation, a document which deprived [him] of all power and practically took away the franchise from the Hawaiian race.
C R E S S T / U C L A
History ExplanationScoring Rubric
1. General impression of content quality2. Principles or concepts (DD)3. Prior knowledge (DD)4. Use of available resources (DD)5. Misconceptions (DD)6. Argument (DD?)7. English mechanics (DI)
C R E S S T / U C L A
Mathematics Explanation
Task (4th or 5th Grade) Imagine a person from a television station has asked you to give a demonstration on TV. You will be on a show to help other students learn about maths. You are asked to explain everything 10-year-old students should know about fractions.
Below are some questions you should try to answer. These are questions that students in the TV audience will ask you.
For each question you should draw as many pictures as you can to show what you mean. Then write down what you would say about your pictures on TV. Use as many words and pictures as you need.
What is a fraction? Why are there two numbers in a fraction? How many fractions are there between 0 and 1? How many fractions are equal to 1/2? What other important ideas should students know about fractions? Show how you would explain these ideas. Use as many pictures and words as you need.
C R E S S T / U C L A
Scoring for Maths Task
Principles
Prior knowledge
Resources
Misconceptions
Argument/explanation
C R E S S T / U C L A
Technology for What?
Most available technology-based assessment promotes efficiency rather than expands the boundaries for measurement of learning
Tendency to limit attention to data processing requirements
We should use technology to extend our understanding of student accomplishment and program and school quality
C R E S S T / U C L A
Template 2: Knowledge Representation to Assess Content Knowledge and
Problem Solving
Same tasks for content knowledge
Responses not essay but representation of relationships, hierarchies
Scored by expert maps
Efficient but expands breadth and depth of knowledge measurement
C R E S S T / U C L A
History: U.S. Depression (CK)
C R E S S T / U C L A
Genetics: High Performance (CK)
C R E S S T / U C L A
Bicycle Pump—High Performance (PS)
C R E S S T / U C L A
Integrating Knowledge Map with Web Search Strategy
C R E S S T / U C L A
MAIN Story: Integrating Formative and External Assessments
Authoring systems (computer-supported guidance) to help teachers create and share various sorts of assessments intended to measure goals included, not covered by, or that need deeper attention than given in external measures
C R E S S T / U C L A
Strategy to Link External and Formative Assessments
Cognitive demands (domain independent) and content knowledge and strategies (domain dependent)
Authoring system allows teachers to assess goals not externally measured, or to connect their assessments to external measures but in a more contextualized setting
C R E S S T / U C L A
How It Works
Teacher will get guidance, using research-based templates
What is the purpose of the test? What is to be measured? What content? What conditions? What intellectual skills?
Scoring rubrics (based on expert performance) will be provided but can be edited
Graphical representation, simulations, and essays are current options
C R E S S T / U C L A
Assessment Authoring Benefits
Common floor on assessments created by teachers
Systems start with easy, fixed formats related to learning demands, and as teacher sophistication develops, move to more choices (mix and match)
Will allow summaries of student work bubbling up from teachers’ formative assessments to validate the external scores
Supports collaboration across different teachers
C R E S S T / U C L A
Authoring Systems Issues
Scored work made public (within the school, with privacy provisions, and among schools)
Success depends upon teacher subject matter knowledge, access to needed information, and sharing
Success may depend on the realistic link to external examinations
Generation of paper- or computer-based tasks
C R E S S T / U C L A
Minimum Requirements
Infrastructure
Capacity—subject matter
Orientation to learning and to results
Congruence with external mandates
Availability of smart tools
Lead to a culture shift
C R E S S T / U C L A
Distance Learning Applications
High-quality performance demanded
No bottleneck in scoring
Basis of comparing courses
Generates online assessments with instant scoring, feedback to student and instructor
Aggregates student work
C R E S S T / U C L A
Window on the Development of Problem-Solving Template 2—
Author Screen Assessment Purpose(s)
Diagnostic, readiness monitoring, certification
Scenario Context, constraints, situation
Problem Characteristics Fix, change usual sequence, improvise step(s), combination
Problem Identification Menu Stated, embedded, multiply masked, barriers, inconsistent
data from multiple sources, time bound, partial identification, prior knowledge requirements
C R E S S T / U C L A
Author Screen Template 2 (Cont’d)
Macro Planning Menu Explicit courses of action, problem subdivision, backup
strategies, help seeking, mix of domain-independent and domain-dependent cognitive strategies
Trial and Feedback Menu Data capture of process, process feedback, help,
iteration
Solution(s) Convergent (right answer), multiple acceptable, partially
acceptable, divergent—with scoring criteria, sequential, mixed
C R E S S T / U C L A
Sample Examinee Screen Components
Scenario
Problem
Information acquisition
Macro strategy
Micro strategies (domain specific)
Solution trials
Report of performance
C R E S S T / U C L A
Sample Examiner Information
Time spent on problem
Trials to criterion
Help access
Solution paths
Generalizability of solution
Acceptability of solution
Likely explanation for errors (e.g., lack of prior knowledge)
Metric (standards for performance)
C R E S S T / U C L A
Summary
Technology-based assessment needs to extend what we can do
Authoring systems can help teachers design better, more sensitive tests and projects
Technology can help us share findings
Technology-based assessment requires the same evidence of technical quality
Demand evidence, not business claims, before you buy