Methods in Perception Researchaix1.uottawa.ca/.../Slides02_Chp1_PerceptualMeasurement_W16_CL… ·...

1

Chapter 1: Fundamentals of Sensory Perception

Methods in Perception Research

Methods in Perception

• Qualitative: Getting the big picture.

• Quantitative: Understanding the details

• Threshold-seeking methods

• Magnitude estimation

• Many others, often similar to those used in other areas of psych.

2

Qualitative Observation

• “Thatcher Illusion”

• Qualitative observation uncovered the phenomenon

• Much quantitative work now trying to find out why it happens & what it means

3

Schwaninger, A., Carbon, C.C., & Leder, H. (2003). Expert face processing: Specialization and constraints. In G. Schwarzer & H. Leder (Eds.), Development of Face Processing, pp. 81-97. Göttingen: Hogrefe.

Qualitative Methods in Perception

• Also called phenomenological or naturalistic observation methods

• Relatively non-systematic observation of a given perceptual phenomenon or environment (e.g., an illusion)

• Yields a verbal description of one’s observations, (possibly with some simple numerical assessment)

• First step in any study of any perceptual phenomenon. Gives the “big picture”

4

Qualitative Methods in Perception

• Example: Famous perception researcher Jan Purkinje noticed that his flower bed looked light red/dark green during the day but dark red/light green at twilight

• This phenomenological observation led to the hypothesis of 2 visual systems, & ultimately to an understanding of the functions of rods and cones

5

6

Quantitative Methods

7

Quantitative Methods in Perception

• Threshold seeking methods measure a physical quantity representing a limit of perceptual ability (i.e., a threshold)

• Measured in physical units (meters, decibels, parts-per-million, candelas of light, etc.)

• Absolute threshold - smallest detectable physical quantity (e.g., 2 dB, 3.57 grams...)

• Difference threshold - smallest detectable difference between two physical quantities

8

Quantitative Methods in Perception

• Thresholds are defined for a given level of response accuracy

• Typically we speak of the “50% threshold”, meaning the physical quantity detectable (absolute threshold) or the physical difference detectable (difference threshold) 50% of the time

• But a threshold can be defined for any level of accuracy (e.g., 75%, 83.6%...)

9

Threshold-seeking Methods

• Classic methods (Fechner, 1850’s)

• Method of adjustment: Quick and dirty

• Method of limits: Easy on observer, fairly fast and accurate

• Method of constant stimuli: Very slow but very accurate

• Adaptive methods: Fast, very accurate, but can be difficult for untrained observers

10

Method of Adjustment

• Stimulus intensity is adjusted (usually by the observer) continuously until observer says he can just detect it

• Threshold is point to which observer adjusts the intensity

• Repeated trials averaged for threshold

• Fast, but not always accurate, due to inherently subjective nature of adjustment

11

Method of Adjustment

12

0.90.80.70.60.50.40.30.20.1Photometer Reading: cd/m2

Instructions: Adjust the intensity of the light using the slider until you can just barely see it

Method of Limits• Stimuli of different intensities presented in

ascending and descending order

• Observer responds to whether she perceived the stimulus

• Cross-over point (between “yes, I see it” and “no, I don’t”) is the threshold for a sequence

• Average of cross-over points from several ascending and descending sequences is taken to obtain final threshold.

13

Method of Limits (descending sequence)

14

0.90.80.70.60.50.40.30.2Photometer Reading: cd/m2

Instructions: For each light intensity, indicate whether you can detect it.

Method of Limits (ascending sequence)

15

0.90.80.70.60.50.40.30.2Photometer Reading: cd/m2

Instructions: For each light intensity, indicate whether you can detect it.

Example Data From Method of Limits

• Why ascending and descending sequences?

• Why different starting points?

16

Method of Constant Stimuli

• 5 to 9 stimuli of different intensities are presented many times each, in random order

• The intensities must span the threshold, so must know approx. where it is a priori.

• Multiple trials (often 100’s) of each intensity are presented

• Threshold is the intensity that results in detection in 50% of trials

17

Method of Constant Stimuli

18

Photometer Readings

Instructions: For each light intensity, indicate whether you can detect it.

0.40.70.40.90.40.20.50.50.5...

0.40.40.90.40.20.50.50.7cd/m2

QuestionHere are some data from a participant in the MoCS. Can you estimate her 50% threshold?

Stimulus Intensity cd/m2 % of stims detected

0.2 0

0.3 0

0.4 0.1

0.5 0.4

0.6 0.7

0.7 0.8

0.8 0.9

0.9 0.9

19

The Psychometric Function

20

.2 .3 .4 .5 .6 .7 .8 .9 (Stimulus Intensity cd/m2)

Calculating 50% Threshold Intensity

• To calculate thresholds we use curve-fitting techniques to fit a sigmoidal (=s-shaped) function to the data (green dots).

• This is called a psychometric function (grey line). It links physical stimulus intensity to performance

• Debate exists over which kind of function--Cumulative Normal, Weibull, Logistic, etc.--is theoretically best, but in practice differences are minor

21

Math Review: Logs

• Remember exponents? 22 = 4

• Logarithms are the opposite log2(4) = 2

• “What exponent will turn the base--usually 2, 10 or e--into the operand?”

• What is the log10(100)? log2(8)?

• Khan Academy: http://tinyurl.com/7zy5wu7

22

Math Review: e (or “Euler’s Number”)• A mysterious constant that just pops up

everywhere in nature. e ≈ 2.71828...

• loge is called the “natural logarithm”, often symbolized as ln

• One also sees ex, where x can be quite a complex expression. “exp(x)” is also used.

• Learn to use your calculator to do logarithms and work with e

23

Math Review: Inverse Functions

• If a function F takes input X and returns Y then inverse F takes input Y and returns X

• Example: Y = 2X inverts to X = ½Y

• The Weibull: y = 1.0 - exp(-(x/b)a)

• The inverse Weibull x = b(-ln(1-y))(1/a)

24

Math Review: Free Parameters

• Free parameter: Part of a math function that is adjusted so that the function fits data

• Example: The intercept (a) and slope (b) of a line are free parameters when fitting a regression line: y = a + bx;

y = 0 +1x

y = 10 +1x

y = 0 +2x

25

Math Review: Free Parameters

• In the Weibull (and its inverse) a and b are free parameters.

• a is the “offset” (in this example a = 1 in all cases)

• b is the “slope” (here varying from 0.5 to 5.0)

y = 1.0 - exp(-(x/5.0)1.0)

y = 1.0 - exp(-(x/1.5)1.0)

y = 1.0 - exp(-(x/1.0)1.0)

y = 1.0 - exp(-(x/0.5)1.0)

26

The Weibull Function

• x is stimulus intensity (in some positive physical unit)

• y is predicted probability of stimulus detection (from 0 to 1)

• a is the “offset” and b is the “slope”

• a and b are free parameters whose values are chosen so that the curve best fits the data. These values are determined by curve-fitting algorithms whose details are beyond the scope of the course.

y = 1.0 - exp(-(x/b)a)

27

Calculating 50% Threshold Intensity Via Inverse Weibull

• To figure out the threshold, we need to figure out what the right values of a and b are for the Weibull that best fits our data.

• We will then enter the percentage threshold we are seeking (e.g., .5 for 50% threshold) into the inverse Weibull (above) to determine the associated stimulus intensity x

x = b(-ln(1-y))(1/a)

28

Calculating 50% Threshold Intensity• Open the excel file "WeibullFit.xlsx"

• Enter your data in two columns: X (stimulus intensity) in column A Y (proportion detected) in column B

• Click the Tools menu and select "Solver…". Note that you may have to first activate the solver. See the document on the class website called “Loading MS Solver Add”.

• In the solver window, click “Options”, then enter “10” in “max time”, then click “Okay”.

• Click "Solve" in the window that appears. After a moment, the Fit Parameters will appear in cells G2 (slope, or A) and G3 (offset, or B)

• So for our data, the inverse Weibull isx = .603 (-ln (1-y) )(1/4.08)

• Plug in the desired probability of 0.5 and to a threshold of 0.5512 cd/m2

29

0"

0.1"

0.2"

0.3"

0.4"

0.5"

0.6"

0.7"

0.8"

0.9"

1"

0.00" 0.20" 0.40" 0.60" 0.80" 1.00"

Y"(Data)"

X"

"Data"

"Fit"

Inverse Weibull step-by-step

x = .603 (-ln (1-0.5) )(1/4.08)

x = .603 (.6931)(1/4.08)

x = .603 (.6931)(.245)

x = .603 (.9141)

x = .5512

30

0"

0.1"

0.2"

0.3"

0.4"

0.5"

0.6"

0.7"

0.8"

0.9"

1"

0.00" 0.20" 0.40" 0.60" 0.80" 1.00"

Y"(Data)"

X"

"Data"

"Fit"

Inverse Weibull step-by-step

x = .603 (-ln (1-0.5) )(1/4.08)

x = .603 (.6931)(1/4.08)

x = .603 (.6931)(.245)

x = .603 (.9141)

x = .5512

31

Example for Self-TestHere are some data from a participant in the MoCS. What is her 50% threshold? What about her 83% threshold?

Stimulus Intensity % of stims detected

0 0

1 0.05

2 0.15

3 0.45

4 0.75

5 0.85

6 0.95

7 0.95

32

50% =c.bj83% = d.fc

Questions

• What is a “free parameter”?

• What is log10(10000)?

33

Adaptive Methods• Examples: Staircase, QUEST, PEST, etc.

• Stimulus first presented at an arbitrary level

• If observer perceives it, intensity is reduced by a predetermined step-size

• If observer does not perceive stimulus, intensity is increased

• This is repeated until several reversals are obtained.

• Threshold is average of reversal points.

34

0

100

200

300

400

500

600

700

N N N N N N Y Y Y N N N Y Y Y N N Y Y N

Observer Response “Do you see it?”

Stim

ulus

Inte

nsity

Reversal Points

Threshold = (700 + 400 + 700 + 400 + 600 + 400) / 6 = 533.33

35

0

100

200

300

400

500

600

700

N N N N N N Y Y Y N N N Y Y Y N N Y N N

36

But why is it called the “Staircase Method”?

0

100

200

300

400

500

600

700

N N N N N Y Y Y N N N N Y Y Y N N Y N Y

Observer Response “Do you see it?”

Stim

ulus

Inte

nsity

37

Calculate the threshold for this run by this observer.

Example for Self-test ebh.f

Adaptive Methods• Similar to method of limits, but we don’t run the same

series of levels each time, instead “adapting” each series based on previous performance.

• A number of specific procedures exist, which modify how the adaptation is done (size of steps, etc.): Staircase, QUEST, PEST, etc.

• Very efficient, because almost all trials are close to the threshold and therefore informative.

• But best used with trained psychophysical observers (can be frustrating for untrained Ss)

38

Forced Choice Variations

• Some threshold-seeking methods can be hampered by response bias.

• Some participants have a lax criterion and tend to say “yes, I see it” a lot.

• Some participants have a strict criterion and tend to say “no, I don’t see it” a lot.

• One way to mitigate this problem is to use forced-choice methods.

39

X-Alternative Forced Choice Variations

• In a forced-choice task, a participant is given several options to choose from: One contains the stimulus and the others don’t.

• The participant is asked, for example: “Which of the two boxes has a light in it?”

• Note that a forced choice method is a modification of (addition to) the threshold-seeking methods we’ve looked at so far.

40

X-Alternative Forced Choice Variations

• Any number of alternatives can be offered, but usually 2 to 8 are given.

• A “2AFC” is a “two-alternative forced choice” procedure, for example.

• The number of alternatives affects the “chance level performance” (= 100% / A)

41

2AFC Method of Limits (descending sequence)

42

0.90.80.70.60.50.40.30.2Photometer Reading: cd/m2

Instructions: For each light intensity, indicate which side it’s on.

Threshold-finding With Other Senses

• The same methods can be used with a wide variety of sensory qualities.

• Different physical units are used depending on the modality:

• Sound amplitude (decibels)

• Touch pressure (pascals)

• Smell/Taste intensity (parts-per-billion)

43

Questions

• What is an absolute threshold?

• Name several methods for measuring absolute thresholds.

• What is an xAFC procedure? Why use one?

• Describe generally how the method of limits works.

44

Difference Threshold• Smallest intensity difference between two

stimuli a person can detect

• Produces a “Just Noticeable Difference” (JND) in degree of subjective sensation

• Same psychophysical methods are used as for absolute threshold

• As magnitude of stimulus increases, so does difference threshold (∆I)

45

Weber’s Law

• Weber’s Law describes the relationship between stimulus intensity (I) and difference threshold (∆I) as follows ∆I / I = K or ∆I = I × K

• This holds true for many senses and many physical quantities across a wide range of moderate intensity levels (but see Steven’s Power Law)

46

Weber’s Law

47

48

Method of Adjustment for Difference Threshold

49

0.90.80.70.60.50.40.30.20.1

Photometer Readings

cd/m2

Instructions: Adjust the intensity of the test light using the slider until you can just barely see the difference between it and the standard light

0.6 cd/m2Standard Light:

Test Light:

2AFC Method of Limits for ∆I

50

Instructions: For each set of lights, indicate which pair (right or left) differ.

• Q: If your Weber fraction (k) is .05, and the standard light’s intensity (Is) is 20, what level will the test light intensity (It)have to be raised to in order for it to be just noticeably different from the standard?

• A: It will have to be higher by the ∆I; That is,

It = Is + ∆I where ∆I = Is × k

∴ It = Is + (Is × k)

It = 20 + (20 × .05) = 21

20 It

Standard Light(Constant)

Test Light(Adjustable)

51

• Q: If your Weber fraction (k) is .20, and the standard light’s intensity (Is) is 50, what level will the test light intensity (It)have to be raised to in order for it to be just noticeably different from the standard?

50 It

Standard Light(Constant)

Test Light(Adjustable)

52

Example for Self-Testfj.j

Questions

• A participant can just tell the difference between lights of 100 cd/m2 and 112 cd/m2. Therefore, she should be able to just tell the difference between 200 cd/m2 and ____ cd/m2.

• What is this participant’s Weber fraction for light intensity?

53

Beyond the Threshold…

Beyond Thresholds

• What do we know about the relationship between subjective sensation and objective intensity at levels above threshold?

• Classic work done by Fechner, who derived Fechner’s Law from Weber’s work.

• This has since been largely supplanted by Steven’s Power Law.

55

Fechner’s Law

• From Weber’s findings, Fechner derived the idea that subjective sensation (S) related to stimulus intensity according to: S = k × ln(I/I0)

• k = empirically-determined free parameter

• Recall that “ln” means “log to base e”

• I = stimulus intensity

• I0 = stimulus intensity at absolute threshold

56

Fechner’s Assumption

• Fechner assumed that each time you went up by one difference threshold (or, subjectively, one JND), that that related to a an equal jump in subjective intensity.

• Intuitively appealing, but turns out to be wrong, as Stevens later showed.

57

Fechner’s Assumption

58

Fechner’s AssumptionSu

bjec

tive

Sens

atio

n (S

)

∆I ∆I ∆I

Objective Stimulus Intensity (I)

59

Fechner’s Law

• Works fairly well for some modalities (loudness, weight), but not others (electric shock) that show response expansion

• More closely models the response of individual neurones than people.

• Still, your music player’s volume control is modelled after Weber’s and Fecher’s laws

60

Magnitude Estimation

• Technique pioneered by Stevens to examine the relationship between subjective perception and objective stimulus intensity at easily perceived levels

• All stimuli are well above threshold

• Observer is given a standard stimulus and a value for its intensity (e.g., “see this light? this is a ‘5’.” )

• Observer assigns numbers to the test stimuli relative to the standard (e.g., “that looks about twice as bright, I’ll call it a ‘10’. ”);

61

Magnitude Estimation

• Relationship between intensity and perception usually shows either:

• Response compression: As intensity increases, the perceived magnitude increases more slowly than the intensity

• Response expansion: As intensity increases, the perceived magnitude increases more quickly than the intensity

62

Magnitude Estimation• Relationship between

intensity and perceived magnitude is a power function

• Stevens’ Power Law

• S = k × Ib

(amazingly simple!)

• Note how for b < 1, we get an approximation of Fechner’s law (red line)

63

Magnitude Estimation• S = k × Ib

• S = perceived magnitude

• I = physical intensity

• k and b are empirically-determined free parameters

• Note that k is not the Weber Fraction

64

The 3 Laws of Psychophysics

• Weber’s law relates two physical units: Standard stimulus intensity and difference threshold

• Fechner’s law relates a physical unit (stimulus intensity) with subjective sensation (or so he thought).

• The above two derive from one another directly, and are sometimes called the Weber-Fechner law

• Steven’s Power Law expands on Fechner’s Law, covering stimuli that show response expansion as well as more closely modelling human responses.

65

Questions

Steven’s Power Law: S = k × Ib

Given that for bitterness b=2 and k=2, what will sensory magnitude (S) be for stimulus intensity (I) of 2 ppm? What if I is doubled to 4 ppm? What does this mean about the relationship between stimulus intensity and sensory magnitude in this case?

66

Sensitivity & Signal Detection Theory

67

Sensitivity & Signal Detection Theory

• Sensitivity (d’)

• Criterion (c)

• Will spend some time on this, as it is used in many fields (though with different jargon): • Diagnostics• Inferential statistics• Human factors engineering

68

Absolute Thresholds Ain’t So Absolute

• Thresholds shift for many reasons unrelated to actual sensitivity.

• An especially problematic factor is criterion (tendency to say “yes” a lot or “no” a lot).

• For example, an individual doing Method of Limits could just say “yes” to all stimuli (lax criterion) and look like he’s incredibly sensitive!

• xAFC methods are one way to deal with this, but another is to measure Sensitivity instead of threshold.

69

Sensitivity

• Sensitivity is symbolized d' (“dee-prime”).

• Measure of one’s ability to detect a given signal (= a stimulus), usually at low intensity.

• Arises from Signal Detection Theory (more later)

• How might we measure sensitivity?

70

(a bad way of) Measuring Sensitivity

• Present stimulus 100 times and note % of times participant says he detects it?

• Problem: P who says “yes” all the time will do very well. (the very problem we are trying to avoid!)

• Need measure that reflects ability to discriminate between “signal present” and “signal absent”

71

(a good way of) Measuring Sensitivity

• Present stimulus (signal) on only half of the trials, (test trials).

• Present no stimulus (noise) on other half of trials, (catch trials).

• Note that unlike threshold experiments, we present only one stimulus at one intensity.

• Many trials are presented. For each, the participant says “yes, the stimulus is there” or “no, it isn’t”.

72

Four Possible Results on Each Trial

Present Absent

Yes, I see it

Hit False Alarm

No, I don’t

MissCorrect

Rejection

The stimulus is really...

Part

icip

ant

says

...

73

Also Known As...

Present Absent

Yes, it’s there

True PositiveFalse Positive(type I error)

No, it’s not

False Negative(type II error) True Negative

The difference is really...

Stat

istic

al t

est

says

...

74

Present (n =100)

Absent (n=100)

Yes, I see it 90 20

No, I don’t 10 80

The stimulus is really...

Part

icip

ant

says

...

Present (n =100)

Absent (n=100)

Yes, I see it 80 10

No, I don’t 20 90

The stimulus is really...

Part

icip

ant

says

...Present

(n =100)Absent

(n=100)

Yes, I see it 90 30

No, I don’t 10 70

The stimulus is really...

Part

icip

ant

says

...

Albert Benny

Claire

75

Some Example Results

Sensitivity• The results of such an experiment yield:

proportion of hits (Ph= Nhits / Ntesttrials)proportion of FAs (Pfa= Nfa / Ncatchtrials)

• For example, for Albert: Ph= 90 / 100 = .9Pfa= 20 / 100 = .2

• Note that we could calc proportions of misses and correct rejections too, but these are redundant (Pm = 1-Ph; Pcr = 1-Pfa)

76

Questions

• What is Benny’s proportion of hits (Ph)?

• What is his proportion of FA’s (Pfa)?

• How do sensitivity experiments differ in method from threshold-seeking methods?

77

Sensitivity

• Perfect participant: Ph = 1, Pfa= 0

• Participant just guessing: Ph = .5, Pfa= .5

• Worst possible participant (perfectly backwards) : Ph = 0, Pfa= 1

• In calculating sensitivity, we want to reward hits and punish FAs, so we could just use “Basic Sensitivity”: BS = Ph - Pfa

78

Sensitivity

• BS equals:

• Perfect participant: Ph of 1 - PFA of 0 = 1

• Participant guessing: Ph of .5 - PFA of .5 = 0

• Backward participant: Ph of 0 - PFA of 1 = -1

• So BS seems to work. However, for obscure statistical reasons, BS is, well, B.S.

79

• Instead we calculated' = z(Ph) - z(PFA)

• Converting the proportions of hits and FAs to z-scores yields a more valid result.

• d’ is measured in standard deviation units.

• How to calculate the z scores?

• In Excel, use norminv(P, 0, 1)

• Table A5.1 from MacMillan & Creelman

• Or the “unit normal” table from any stats textbook.

80

Careful! This means function z, do NOT multiply z by PFA)

Normal distribution for finding d' (based on Macmillan & Creelman (2005), Signal Detection: A User’s Guide)

81

Sensitivity

• d' = z(Ph) - z(Pfa)

• Albert: Ph = .9 ∴ z = 1.28; Pfa = .2 ∴ z = -0.84∴ d’ = 1.28 - (-0.84) = 2.12

• Ben: Ph = .8 ∴ z = 0.84; Pfa = .1 ∴ z = -1.28∴ d’ = 0.84- (-1.28) = 2.12

• Claire: Ph = .9 ∴ z = 1.28; Pfa = .3 ∴ z = -0.52∴ d’ = 1.28 - (-0.52) = 1.8

82

Link back to data (#91)

Zero and One

• What to do with proportions of 0 and 1?

• These yield z scores of –∞ and +∞• There are many ways of getting around this

(MacMillan & Creelman, 2005, chp. 1). For our purposes, just substitute values of 0.01 and 0.99, respectively.

83

Present (n =100)

Absent (n=100)

Yes, I see it 0 0

No, I don’t 100 100

The stimulus is really...

Part

icip

ant

says

...

Present (n =100)

Absent (n=100)

Yes, I see it 100 100

No, I don’t 0 0

The stimulus is really...

Part

icip

ant

says

...

Present (n =100)

Absent (n=100)

Yes, I see it 100 0

No, I don’t 0 100

The stimulus is really...

Part

icip

ant

says

...

Easy Eddie

DoubtingDan

FlawlessFran

84

Sensitivity

• d' = z(Ph) - z(PFA)

• Dan: Ph = 0 ∴ z = -2.33; PFA = 0 ∴ z = -2.33∴ d’ = -2.33 - (-2.33) = 0 (usual minimum)

• Eddie: Ph = 1 ∴ z = 2.33; PFA = 1 ∴ z = 2.33∴ d’ = 2.33 - 2.33 = 0 (usual minimum)

• Fran: Ph = 1 ∴ z = 2.33; PFA = 0 ∴ z = -2.33∴ d’ = 2.33 - (-2.33) = 4.66 (conventional max.)

85

Questions

• Example for self-test: Zack does a sensitivity experiment with 20 test trials and 20 catch trials. He gets 10 hits and 5 false alarms.

• What are his Ph and Pfa values?

• What is is d’?

86

Ph = .ejPfa= .be

d’ = .fg

Criterion• The flip side of sensitivity is criterion or

response bias.

• How do we measure a person’s tendency to say yes or no? There are several measures, but the simplest is: c = [z(Ph) + z(Pfa)] / -2

• The lower c, the more a person’s tendency to say “yes”. When c<0, yes > no. When c>0, yes < no.

87

Criterion

• c = [z(Ph) + z(PFA)] / -2

• Albert: Ph = .9 ∴ z = 1.28; PFA = .2 ∴ z = -0.84∴ c = (1.28 - 0.84) / -2 = -.22 [tends to “yes”]

• Ben: Ph = .8 ∴ z = 0.84; PFA = .1 ∴ z = -1.28∴ c = (0.84 -1.28) / -2 = .22 [tends to “no”]

• Claire: Ph = .9 ∴ z = 1.28; PFA = .3 ∴ z = -0.52∴ c = (1.28 - 0.52) / -2 = -.38 [tends to “yes”]

88

Link back to data (#91)

Isosensitivity CurvesA range of Ph and Pfa values can produce the same d’.

Ph Pfa z(Ph) z(Pfa) d’ c

0.8 0.4 0.84 -0.25 1.09 -0.295

0.6 0.2 0.25 -0.84 1.09 0.295

0.35 0.07 -0.39 -1.48 1.09 0.935

This occurs when sensitivity remains the same, but criterion shifts

89

(a.k.a. “ROC curves”)

Isosensitivity CurvesPlotting results for different criterion levels at the same sensitivity yields an isosensitivity curve (aka, ROC curve)

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

HitRate

FalseAlarmRate

ROCCurved'=1.09

d’ = 1.09, criterion = -0.30

d’ = 1.09, criterion = 0.94

d’ = 1.09, criterion = 0.30

90

Hit

Rat

e (P

h)

False Alarm Rate (Pfa)

Miss R

ate (Pm )

Correct Rejection Rate (Pcr)

1.0

.5

0

0.0

.5

1.00 0.5 1.0

1.0 0.5 0

91

Why Does Criterion Shift?

• Many reasons, but an important one is the payoff matrix.

• If the cost of a false alarm, or reward for a correct rejection, is great, one will tend to make one’s criterion stricter (more “no”)

• If the cost of a miss, or reward for a hit, is raised, one will tend to make one’s criterion laxer (more “yes”)

92

Example of a Pay-off Matrix Leading to a Lax Criterion

Cancer is...

Present (n =100)

Absent (n=100)

Yes, I see it

Patient Saved Unnecessary Additional Testing

No, I don’t

Patient Dies

Patient Goes Home

Radiologist Says...

93

Present (n =100)

Absent (n=100)

Yes, I see it

+100$ -$10

No, I don’t

-$10 +10$

The stimulus is really...

Part

icip

ant

says

...

Present (n =100)

Absent (n=100)

Yes, I see it

+10$ -$10

No, I don’t

-$10 +100$

The stimulus is really...

Part

icip

ant

says

...

Lax Strict

Present (n =100)

Absent (n=100)

Yes, I see it

+10$ -$10

No, I don’t

-$10 +10$

The stimulus is really...

Part

icip

ant

says

...

Neutral

94

Questions

Dr. X tests a new field diagnostic procedure by applying it to 50 individuals known to have an illness. He finds that it correctly labels all 50 of them as ill, whereas his old procedure labeled only 40 of them as such. He concludes that the new procedure is better. Is his conclusion sound? Why or why not?

95

Questions

• Why would one be interested in measuring changes in criterion?

• True or False: As one travels up and to the right along an isosensitivity curve, criterion rises.

• True or False: A higher criterion value indicates a stronger tendency to say “yes, the stimulus is present”

96

Signal Detection Theory

• The concepts of sensitivity and criterion arise out of Signal Detection Theory (SDT)

• SDT suggests that any attempt at detecting a signal (stimulus) has to contend with competing noise

• Noise in this sense is random variations from the environment (e.g., literal noise) or from within the detector (e.g., neuron chatter)

97

SDT

• Example: You’re in the shower and expecting a call.

• Signal: Phone ringing

• Noise: Sound of shower (plus all other sources of sound), as well as your own internal neuron chatter.

• If the phone isn’t ringing, you have just noise. If the phone is ringing, you have signal+noise.

• d’ is essentially a measure of how similar the signal is to the noise for you subjectively.

98

phone + noise

How much it sounds subjectively like a phone

unmistakablynot at all

noise only

Prob

abili

ty

a bit a lotkinda

Probability Distributions of Noise and Signal+Noise

Link back to matrix

99

phone + noise

How much it sounds subjectively like a phone

completelynot at all

noise only

Prob

abili

ty

a bit a lotkinda

Probability That a Given Perceptual Effect is Due to N or S+N

100

phone + noise

How much it sounds subjectively like a phone

completelynot at all

noise only

Prob

abili

ty

a bit a lotkinda

Different Criteria One Can Adopt

Lax

Result: Pfa = .50 Ph = .95 d' = 1.64

101

phone + noise

How much it sounds subjectively like a phone

completelynot at all

noise only

Prob

abili

ty

a bit a lotkinda

Different Criteria One Can Adopt

Strict

Result: Pfa = .05 Ph = .50 d' = 1.64

102

phone + noise

How much it sounds subjectively like a phone

completelynot at all

noise only

Prob

abili

ty

a bit a lotkinda

Increasing d’ means increasing the distance betwen the probability distributions. Note how, with a greater d’, a criterion somewhere in the neutral area will produce very few misses or false alarms.

d’ (large)

103

phone + noise

How much it sounds subjectively like a phone

completelynot at all

noise only

Prob

abili

ty

a bit a lotkinda

Decreasing d’ means decreasing the distance between the probability distributions. Note how, with a smaller d’, many misses and/or false alarms result, regardless of where the criterion is placed.

d’ (small)

104

Questions

• In terms of the SDT, what would be the effect of turning up the phone volume on the probability curves? What effect would this have on d’? What about turning down the shower?

• A guard is watching the woods for visible signs of intruders. What are some likely sources of “noise” in his situation?

105

For More Info...

A nice primer: Stanislaw, H., & Todorov, N. (1999). Calculation of signal detection theory measures. Behavioral Research Instruments, Methods and Computers, 31, 137-149.

The bible of SDT: Macmillan, N. A., & Creelman, C. D. (2005).  Detection Theory:  A User's Guide (2nd ed.). Mahwah, N.J.: Lawrence Erlbaum Associates.

106