Linking Learning to Assessments

BJDSOP Faculty Retreat5/16/2012

Linking Learning to Assessment

1. Writing learning objectives2. Relating objectives to assessment 3. Writing assessment questions4. How to analyze knowledge tests for

discrimination

Learning objectives

1. Define an objective.2. How/ when to author3. How to align objectives with

assessments.

Characteristics of objectives

• Specific and focused• Targets performance• Realistic to achieve• Can be measured and validated• Time-bound with a deadline

Learning objectives…

• … describe the intended result of instruction.

LWBAT

Outcomes• recall, identify, choose, • solve, • calculate• apply therapeutic concepts to

clinical scenarios

Iterative process

Lect

ure/

lab

“Begin with the end

in mind”Stephen CoveyThe Seven Habits of Highly Effective People

•We need to know what we want the end result to be before we plan.

Write your test questions first thenwrite your objectives

Questions already written?

•Rewrite objectives to align with what is required of the students on the assessments

Rationale1. Alignment 2. Focuses attention on what is

most important* 3. Promotes continuous

improvement

Consider these….

Example 1• Question– Which statement is correct…

• Objective: –Student will be able to identify..

Example 2• Refer to the case above.• What is the most appropriate

therapy at this time?

• Objective: –Student will be able to select/…

Work smart• What (are) the KSAs, being

assessed by your test questions?

• Restate those as your objectives in LWBAT terms…..

Objectives RECAP

1. What is a learning objective?2. When do we right them? 3. Why?4. How do we align objectives

with assessments?

What is a good question?

1. Good form

Bad format DO NOT COPY

Good format – please copy

Factual recall format

Better

Basic Science examples

More BS examples

Vignettes NBME • “..we believe vignette items are generally

more appropriate ..”–test application of knowledge to

patient situations

–pose appropriate clinical challenges

Non vignette

Short Vignette

Long Vignette

What is a good question?

1. Good form2. Best practices

www.nbme.org/

NBME Guidelines

Best Practices•No T F•MC with 4-5 good options •1 correct answer preferred

Testwiseness BAD• Grammatical cues • Absolute terms • Long correct answer•Word repeats

Irrelevant Difficulty BAD

• Options are long, complicated• Numeric data not stated

consistently• Vague – “rarely”• Avoid all/none of the above• Hinged responses

K- type questions • Avoid• Re

Good questions• Big stems • ~ Same length distractors (short)• Avoid absolutes / vague terms• Avoid negatively phrased items

What is a good question?

1.Good form2.Best practices3.Performs well

Performance

A good question?1. % correct 2. Item discrimination 3. Distractor performance

Item Analysis

A good question?1. % correct 2. Item discrimination 3. Distractor performance

Test matrix 2010 2011 %

Change

CK13 11 (15.3%)

APP3 5 66.6%

Correct response CK 86

90.5 4.9%

Correct APP62 65.8 6.1%

Bloom’s TaxonomyLevel

1. Knowledge

2. Comprehension

3. Application

4. Analysis

5. Synthesis

6. Evaluation

Less complex

More complex

VERBSBloom’s Level Verbs

1. Knowledge match, recognize, select, compute, define, label, name, describe

2. Comprehension restate, elaborate, identify, explain, paraphrase, summarize

3. Application Apply knowledge, solve problems

4. Analysis outline, draw a diagram, illustrate, discriminate, subdivide

5. Synthesis compare, contrast, organize, generate, design, formulate

6. Evaluation support, interpret, criticize, judge, critique, appraise

Linking Learning to Assessment

1. Writing learning objectives2. Relating objectives to assessment 3. Writing assessment questions4. How to analyze knowledge tests for

discrimination

References

1. Case SM, Swanson DB, Becker DF. Verbosity, window dressing, and red herrings: do they make a better test item? Academic Medicine. 1996;71:528-530.

2. NBME Constructing Written Test Questions for the Basic and Clinical Sciences

3. Haladyna, T. M., & Downing, S. M. (1989a). A taxonomy of multiple-choice item-writing rules. Applied Measurement in Education, 2(1), 37-50.

4. Frisbie, D. A. (1990, April). The evolution of the multiple true-false item format. Paper presented atthe Anual Meeting of the National Council on Measurement in Education, Boston.

• Sands (2002), the “basic precept of course-planning [is]: What do [you] want students to be able to do at the end of the semester?” In other words, course goals and objectives should guide the design of your course rather than technology (Aycock, Garnham, & Kaleta, 2002). Sands’ first principle for developing a blended course is to “work backward from the final course goal…to avoid a counterproductive focus on technology.”