C R E S S T / U C L A Usable Assessment Knowledge: A Design Problem Eva L. Baker International Congress for School Effectiveness and Improvement January

C R E S S T / U C L A

Usable Assessment Knowledge: A Design

Problem

Eva L. Baker

International Congress for School Effectiveness and ImprovementJanuary 6, 2003

Sydney, Australia

UCLA Graduate School of Education & Information StudiesCenter for the Study of Evaluation

National Center for Research on Evaluation, Standards, and Student Testing

With support from the Education Institute of Science, U.S. Department of Education


Knowledge to Support Educational Improvement

UnusablePlentiful

Usable Scarce

Useful Rare


Usable and Useful Knowledge

Usable Knowledge

In a form that can be understood

In a form that can be applied

Timed appropriately

May cause rethinking of the problem

Useful Knowledge

Rethinking indicates a new solution path

Adapted to situation

Sufficient to guide solution

Improved outcomes occur as a result


Why Are Some Schools Successful in Using

Knowledge?

Focus on learning (students and adults)

Constant use of appropriate information (formal and informal)

Focus on feedback and change

Public display and exchange

Pride in outcomes of students and place

Select knowledge to foster these ends


Assessment Knowledgein the Service of Reform

Usable?

Useful?


Assessment: Historical Transformations

1. Assessment = measurement and interpretation of a sample of performance in the desired area of learning

2. Assessment (test) score = learning

Activities not contributing should be dropped

Coincidence = causality

3. Assessment is the best intervention

Cost/ Effectiveness

(Useful)

(Usable)

(Unusable)


Types of Assessment Knowledge

Purposes for assessment (Why? For whom?)

What to assess

Whom to assess

How to assess (design and procedures)

How to interpret and report results

How to determine if results are trustworthy (validity) for the purpose(s)


Key Design Principles for Useful Assessment

Assessment systems that start with thinking skills and apply them to content domains support

Coherent, sustained learning

Spiral teaching

Transfer (application to new or unforeseen situations)


CRESST Assessment Models

Research-based

Focus on cognition and learning

Reusable and cost-sensitive

Operationalized in models and templates


Model-Based Assessment Families of Cognitive

Demands

ContentUnderstanding

ProblemSolving

Teamwork andCollaboration

MetacognitionCommunication

Learning


Usable Model-Based Assessment Design

Specifications and “Templates” for assessment protocols, including scoring

Templates that allow common design approaches to be used, e.g., primary sources

Two template examples for the model of deep understanding of content

Explanation

Graphical representation of relationships


Specifications of Model for Content Understanding

Primary source materials in each domain

Student required to integrate prior knowledge and principles to succeed

Scored by using expert model in subject matter


Content UnderstandingTemplate #1 Explanation

An array of primary source materials

A prompt that asks for an explanation in context

Constructed (written) answer

Evaluated by means of a scoring rubric


Hawaiian History Writing

Assignment: BayonetConstitution

Be sure to show the relationships among your ideas and facts.

Your essay should be based on two major sources:

1. The general concepts and specific facts you know about Hawaiian history, and especially what you know about the period of the Bayonet Constitution.

2. What you have learned from the readings yesterday.

Imagine you are in a class that has been studying Hawaiian history. One ofyour friends, who is a new student in the class, has missed all the classes.Recently, your class began studying the Bayonet Constitution. Your friend isvery interested in this topic and asks you to explain everything that you havelearned about it.

Write an essay explaining the most important ideas you want your friend tounderstand. Include what you have already learned in class about Hawaiianhistory, and what you have learned from the texts you have just read. Whileyou write, think about what Thurston and Liliuokalani said about the BayonetConstitution, and what is shown in the other materials.


EXCERPTS from HAWAIIAN HISTORYPRIMARY SOURCE DOCUMENTS

LILIUOKALANI

For many years our sovereigns had welcomed the advice of American residents who had established industries on the Islands. As they becamewealthy, their greed and their love of power increased. Although settledamong us, and drawing their wealth from resources, they were alien to usin their customs and ideas, and desired above all things to secure their own personal benefit.

Kalakaua valued the commercial and industrial prosperity of his kingdomhighly. He sought honestly to secure it for every class of people, alien ornative. Kalakaua’s highest desire was to be a true sovereign, the chiefservant of a happy, prosperous, and progressive people.

And now, without any provocation on the part of the king, having maturedtheir plans in secret, the men of foreign birth rose one day en masse, calleda public meeting, and forced the king to sign a constitution of their ownpreparation, a document which deprived [him] of all power and practically took away the franchise from the Hawaiian race.


History ExplanationScoring Rubric

General Impression of Content Quality

Principles or Concepts

Prior Knowledge

Use of Available Resources

Misconceptions (negative)

Argumentation (domain appropriate)

English Mechanics


Mathematics Explanation

Task (9-year-olds) Imagine a person from a television station has asked you to give a demonstration on TV. You will be on a show to help other students learn about math. You are asked to explain everything students your age should know about fractions.

Below are some questions you should try to answer. These are questions that students in the TV audience will ask you.

For each question you should draw as many pictures as you can to show what you mean. Then write down what you would say about your pictures on TV. Use as many words and pictures as you need.

What is a fraction? Why are there two numbers in a fraction? How many fractions are there between 0 and 1? How many fractions are equal to 1/2? What other important ideas should students know about fractions? Show how you would explain these ideas. Use as many pictures and words as you need.


Template #2Knowledge Representation

Key aspects of ideas, supporting facts and views and their relationships

Relationship is explicit

Organizational options

Core and peripheral

Hierarchical

Cause and effect

Chronological

Expert scoring


History


Genetics


Bicycle Pump


Key Design Principles from Knowledge Requirements

Knowing why

Knowing what to assess: content plus cognitive demands (problem solving, communication, learning to learn, teamwork, content knowledge)

Knowing how: to make tests that measure and support transfer (application to other topics and situations)


To Make Assessment Knowledge Useful for

Educators Timing

Having sufficient content knowledge

Having sufficient knowledge of students

Knowing how to combine results and other sources of information

Knowing where to find help and resources

Knowing what to do (more than one option)


Data Interpretation—Design Principles

How to interpret results (in comparison to what?)

How to understand data from sources other than classroom testing

How to integrate sources of test results and form ideas about what to do next


Decision Support System

: allows the integration of coherent information

: allows the identification of conflicting or discordant data

: requires the judgment of the user to assign value



QSP

Creates longitudinal records from external sources

Easy to read icons

Grade books for teachers

Digitized examples of student work

Easy to use queries about relationships

Parent conferencing material


Groups - for disaggregation

Reports - for displaying information graphically

Goals - for monitoring improvement

Gradebook - for keeping tabs on student progress

Digital Portfolio - for examples of student work

Resource Kit - to gather locally important information (bottom-up)

Functions


Communication

The teacher decided to send a QSP Progress

Report to the student’s parents.


Using Assessments and Reflecting on Data Is Hard

Teachers: Fundamental shift from chronological organization to functional organization

From what am I doing?

To what should each learner be doing now? Time and collaboration

Administration: Taking a chance on change


Context for Success of Knowledge-Based Reform

Local ownership of knowledge

Infrastructure and stability

Capacity to investigate

Learning

Congruence or peace with external mandates


Usable Knowledge and Support May Get to Useful

Knowledge

For assessment knowledge to be useful, it depends upon the context, capacity, and communication of the teaching system

For assessment knowledge to be useful to students, it must go to the heart of why, what, and how they learn

Documents

C R E S S T / U C L A Usable Assessment Knowledge: A Design Problem Eva L. Baker International Congress for School Effectiveness and Improvement January