“Problems and Priorities in Pretesting Pondered”

CLEAR 2008 Annual Conference

Anchorage, Alaska

“Problems and Priorities in Pretesting Pondered”

Beth Noeller

Grady Barnhill

Carol O’Byrne

CLEAR 2008 Annual ConferenceAnchorage, Alaska

CLEAR 2008 Annual ConferenceAnchorage, Alaska

All Items are not Created Equal

CLEAR 2008 Annual ConferenceAnchorage, Alaska

Purposes of Pretesting

Gather performance information

Add new content

CLEAR 2008 Annual ConferenceAnchorage, Alaska

What Information do you Gather?

• Performance data♦ Classical and IRT

♦ Difficulty

♦ Discrimination

• Cognitive Level Data

• Content

CLEAR 2008 Annual ConferenceAnchorage, Alaska

Difficulty

• How hard is the item?

• Usually expressed as a %.

• What percent of the candidates get the item correct?

• IRT b-value

CLEAR 2008 Annual ConferenceAnchorage, Alaska

What is a “good” difficulty?

• What is your cut score?

• The best difficulty is right around the cut score.

• Too easy and too hard provide little information.

• Between 40 & 90 for many testing programs

CLEAR 2008 Annual ConferenceAnchorage, Alaska

Discrimination

• How well an item differentiates between those who perform well on the test and those who get the item correct.

• Correlation• Values from -1.0 to +1.0• IRT a-value

CLEAR 2008 Annual ConferenceAnchorage, Alaska

Positive Discrimination Values

• Two people walking together ‘in step’

• Get an item correct and receive a high score on the examination

• Perfect positive relationship +1.0

Item correctness

exam score

CLEAR 2008 Annual ConferenceAnchorage, Alaska

Negative Discrimination Values

• Two people walking in opposite directions

• Get an item correct and receive a low score on the examination

• Perfect negative relationship -1.0

Item correctness

exam score

CLEAR 2008 Annual ConferenceAnchorage, Alaska

What is a “good” discrimination?

• A positive relationship

• The higher the better

• Diversity of the candidate population

• Greater than 0.15

exam score

Item correctness

CLEAR 2008 Annual ConferenceAnchorage, Alaska

Combining Difficulty and Disc.

• Each quality by itself is linear

• Together they can be depicted on a two-dimensional graph

CLEAR 2008 Annual ConferenceAnchorage, Alaska

Nursing

-0.30

-0.20

-0.10

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

0 10 20 30 40 50 60 70 80 90 100

Difficulty

Dis

crim

inat

ion

CLEAR 2008 Annual ConferenceAnchorage, Alaska

Health Related 4 yr +

-0.35

-0.25

-0.15

-0.05

0.05

0.15

0.25

0.35

0.45

0.55

0 10 20 30 40 50 60 70 80 90 100

Difficulty

Dis

crim

inat

ion

CLEAR 2008 Annual ConferenceAnchorage, Alaska

Health Related2 yr

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

0.6

0 10 20 30 40 50 60 70 80 90 100

Difficulty

Dis

crim

inat

ion

CLEAR 2008 Annual ConferenceAnchorage, Alaska

Heath Related 4 yr

-0.45

-0.35

-0.25

-0.15

-0.05

0.05

0.15

0.25

0.35

0.45

0.55

0.65

0 10 20 30 40 50 60 70 80 90 100

Difficulty

Dis

crim

inat

ion

CLEAR 2008 Annual ConferenceAnchorage, Alaska

Health Related6 yrs +

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

0 10 20 30 40 50 60 70 80 90 100

Difficulty

Dis

crim

inat

ion

CLEAR 2008 Annual ConferenceAnchorage, Alaska

Health Related6 yr +

-0.35

-0.25

-0.15

-0.05

0.05

0.15

0.25

0.35

0.45

0 10 20 30 40 50 60 70 80 90 100

Difficulty

Dis

crim

inat

ion

CLEAR 2008 Annual ConferenceAnchorage, Alaska

New Medical

-0.15

-0.05

0.05

0.15

0.25

0.35

0.45

0.55

0 10 20 30 40 50 60 70 80 90 100

Difficulty

Dis

crim

inat

ion

CLEAR 2008 Annual ConferenceAnchorage, Alaska

New Medical

-0.1

0

0.1

0.2

0.3

0.4

0 10 20 30 40 50 60 70 80 90 100

Difficulty

Dis

crim

inat

ion

CLEAR 2008 Annual ConferenceAnchorage, Alaska

Mature Medical

-0.25

-0.15

-0.05

0.05

0.15

0.25

0.35

0.45

0 10 20 30 40 50 60 70 80 90 100

Difficulty

Dis

crim

inat

ion

CLEAR 2008 Annual ConferenceAnchorage, Alaska

Health Related

-0.25

-0.15

-0.05

0.05

0.15

0.25

0.35

0.45

0.55

0.65

0 10 20 30 40 50 60 70 80 90 100

Difficulty

Dis

crim

inat

ion

CLEAR 2008 Annual ConferenceAnchorage, Alaska

Non-Health

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0 10 20 30 40 50 60 70 80 90 100

Difficulty

Dis

crim

inat

ion

CLEAR 2008 Annual ConferenceAnchorage, Alaska

Non-HealthMath

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0 10 20 30 40 50 60 70 80 90 100

Difficulty

Dis

crim

inat

ion

CLEAR 2008 Annual ConferenceAnchorage, Alaska

Professionalnew @ pretesting

-0.15

-0.05

0.05

0.15

0.25

0.35

0.45

0 10 20 30 40 50 60 70 80 90 100

Difficulty

Dis

crim

inat

ion

CLEAR 2008 Annual ConferenceAnchorage, Alaska

CLEAR 2008 Annual ConferenceAnchorage, Alaska

CLEAR 2008 Annual ConferenceAnchorage, Alaska

Professional

-0.15

-0.05

0.05

0.15

0.25

0.35

0.45

0 10 20 30 40 50 60 70 80 90 100

Difficulty

Dis

crim

inat

ion

CLEAR 2008 Annual ConferenceAnchorage, Alaska

No. Pretest Items

Survival %

% neg. disc

% low disc

% easy % hard

2900 44% 10% 33% 2% 11%1000 40% 8% 36% 12% 3%760 58% 6% 24% 9% 4%597 42% 15% 30% 10% 4%922 46% 13% 36% 8% 3%

1325 31% 17% 41% 8% 3%294 55% 5% 32% 15% 3%197 46% 2% 35% 7% 11%800 41% 15% 35% 4% 5%622 35% 13% 44% 5% 4%810 63% 0% 2% 35% 0%430 68% 0% 1% 0% 31%240 56% 3% 28% 5% 8%

1147 58% 5% 22% 6% 9%100 6% 13% 54% 14% 13%46 28% 22% 39% 7% 4%

CLEAR 2008 Annual ConferenceAnchorage, Alaska

Summary

No. Pretest Items

Survival %

% neg. disc

% low disc

% easy % hard

12,190 45% 9% 31% 9% 7%

Range 6-68% 0-22% 1-54% 0-35% 0-31%

CLEAR 2008 Annual ConferenceAnchorage, Alaska

How do you use pretesting to add new content?

• Get the pulse of your item bank

• Inventory of acceptable scored items

• Inventory of unused items

CLEAR 2008 Annual ConferenceAnchorage, Alaska

Scored Items and Test Specs

12%

40%

10%

17%

21%

18%

35%

8%

20%

19%

I

II

III

IV

V

Inner Circle = Content Specifications

Outer Circle = Used Pool

CLEAR 2008 Annual ConferenceAnchorage, Alaska

Pretest Needs

12%

40%

10%

17%

21%

6%

46%

12%

13%

23%

I

II

III

IV

V

Inner Circle = Content Specifications

CLEAR 2008 Annual ConferenceAnchorage, Alaska

Beth’s Contact Information

Beth Noeller, PhD

319-339-3095

beth.noeller@act.org