Using the PADI Framework to Design an Item

PADI Item Design C. Kennedy 3/25/03 1

Using the PADI Framework to Design an Item

?


GLOBE Task Template


Templates store item characteristics

Flow of Inference

Flow of Design

Student Model

Measurement

Model

Evaluation Procedures

Work Product Specs

Presentation Materials & Properties

4 Building Blocks of Instrument Development

Evidence Centered Design of Assessments


Presentation Materials & Properties

Item presented to student

Response is captured


1.Think about the STUDENT MODEL

What do you intend the assessment to measure?

From the DP(s), look at Rationale, Focal KSAs, Additional KSAs to help determine the proficiency(ies) you want to measure.


Then, develop a Student Model (model of student progress in one or more aspects of proficiency—the Student Model Variables).

For a FOSS example, the SM is comprised of:

SMV1, Using Appropriate Tools

SMV2, Graphing

SMV3, Distance, Speed & Acceleration

1.Think about the STUDENT MODEL

See under the hood:


What do you intend the item to measure? Select one or more Student Model Variables for the item.

We want the item to measure:

SMV3, Distance, Speed &Acceleration

(In particular, we want to measure the student’s understanding

of reference points and the distance equation.)

Alternate set of SMVs:

SMV1, Using Appropriate Tools

SMV3, Distance, Speed & Acceleration

2. Think about the MEASUREMENT MODEL


What form should the item and response take? •Dichotomous (one correct answer).

•Partial credit (different valuations (scores) for qualitatively different student responses).

•Rating scale (all rubrics have similar scales and going from a level 2 response to a level 3 response means the same thing for each item).

•Item bundle (linked prompts and responses, e.g. 2 connected items: “1) Here is some data, draw a graph showing the relationship of distance traveled to propeller winds, and 2) explain what the graph shows about the number of winds and the effect on distance”).



What form should the item and response take?

Partial credit – students will construct a response from a word description and graphic. They will show the steps they followed to answer the question.


“One person drove a truck and the other person drove a car. Which vehicle went farther? How much farther?” See under

the hood:


What form should the item and response take?

Alternate structure: A series of dichotomous responses, comprising an item bundle:


1. Show the initial and final reference points for the car [truck].

2. Show the equation for calculating the distance the car [truck] traveled.

3. How far did the car [truck] travel?

4. Which traveled farther?

5. How much farther?


What evaluation procedures will be followed to transform the student response to an observable variable that can be used to estimate proficiency?

Define the scoring rubric(s) to be used in evaluating student work, for example, one for each SMV measured on the item (this step is often done in conjunction with step 1 when defining the Student Model of interest).

3. Think about the EVALUATION PROCEDURES


3. Think about the EVALUATION PROCEDURES

4 Understands speed when time or distance are not held constant

3 Understands speed for simple cases

2 Understands distance

1 Intuitive understanding

0 Unresponsive

(NOTE: The example here was not developed as a scoring guide, but

I’ve used it here for convenience).


3. Think about the EVALUATION PROCEUDRES

An item may be associated with multiple student model variables.

Item A

SMV1

SMV3

The item involves both SMV1 and SMV3. Two scoring guides could be developed and the value of the observable variable could depend on combinations of scores on each of the SMVs.


3. Think about the EVALUATION PROCEUDRES

Multiple responses may comprise an item bundle (even with multiple student model variables!).

Response 1 SMV1

SMV3

The item has several responses, which are conditionally dependent. The value of the observable variable will depend on combinations of responses.

Response 2

Response


Conclusion: Initial steps in item design

1. Think about the Student Model: What do you intend the assessment to measure?

2. Think about the Measurement Model: What do you intend each item to measure?

3. Think about the Evaluation Procedures: How will student responses be evaluated to provide evidence about what students know and can do?

Documents

Using the PADI Framework to Design an Item