18/2/2002 J.Y.Y. Kwong 1

L4 - Assessment Methods (II)SOC325 Behavior at workRef: Spector, Ch. 2 (p.32-36)

18/2/2002 J.Y.Y. Kwong 2

Outline:1. Different types of assessment methods for personnel selection

InterviewsBiographical informationWork samplePsychological testsAssessment center

2. Assessing test reliability and validity

18/2/2002 J.Y.Y. Kwong 3

To have an accurate assessment of eye-sight, we need a good optician:

How do you pick one?

How to pick a good assessment test?

What it takes to be a GOOD test?

18/2/2002 J.Y.Y. Kwong 4

A scientific way to evaluate a test:

A good test should demonstrate:High reliabilityEvidence for its validity

18/2/2002 J.Y.Y. Kwong 5

168 lb

Reliability

A test is reliable when it gives consistent measurements across repeated observations.

170 lb1 min. later

You read175 lb

You read

NO!I’m 168 lb only!

He read Hey!It’s not reliable!

18/2/2002 J.Y.Y. Kwong 6

Reliability

Test-retest reliability: A test has high test-retest reliability when it gives consistent measurement across times.

160 lb

160 lb

160 lb

1 min. later 1 min. later

18/2/2002 J.Y.Y. Kwong 7

Reliability

Inter-rater reliability: A test has high inter-rater reliability when different users (raters) using it give consistent measurements of the same object.

18/2/2002 J.Y.Y. Kwong 8

Reliability

Internal consistency reliability: For tests that use multiple items to give a measurementEach item should be consistently measuring the same variable

Overall, I am satisfied with the teacher’s performance.v.s.

1. The teacher has structured the presentation well.2. The teacher encouraged students’ participation.3. The size of students’ desk was suitable for writing.4. The teacher was responsive to students’ questions.

18/2/2002 J.Y.Y. Kwong 9

Reliability

Internal consistency reliability: A test has high ICR when multiple items give similar or consistent measurement.

1. The teacher was structured the presentation well.2. The teacher encouraged students’ participation.3. The size of students’ desk was suitable for writing.4. The teacher was responsive to students’ questions.

18/2/2002 J.Y.Y. Kwong 10

A good test has high reliability:

Test-retest – across time (same user; same target)Inter-rater – across raters/users (same target; same time)Internal consistency reliability – across items (same target; same user; same time)

18/2/2002 J.Y.Y. Kwong 11

The concepts of validityTarget Instrument

Length Income

Health Ruler

Intelligence O-P scale

Optimism Blood Pressure

Social Status IQ test

18/2/2002 J.Y.Y. Kwong 12

Validity

Is the tool appropriate to measure the KSAO that we are interested in?

A test can be reliable but not valid! (Ruler – Optimism)

A test is valid when there is evidence that the test score reflects what the test is supposed to measureHow do we know whether a ruler or O-P scale really measures optimism?3 types of evidence

Face; Content; Criterion-related validity

18/2/2002 J.Y.Y. Kwong 13

Validity

Face validity: the test looks like what it was designed to measure (subjective).劃一個人 , 一間屋 , 一條路

測你的人際關係能力 .O-P

我很少會感到沮喪我很快便忘記不愉快的事情面對困難時 , 我仍能保持樂觀

18/2/2002 J.Y.Y. Kwong 14

ValidityContent validity: the test adequately covers the content of the variable (subjective).你有成為好男友的質素嗎 ?

別人曾讚頌你清潔整齊你是個有耐性的聆聽者你每天有梳理頭髮的習慣你不挖鼻子

O-P我很少會感到沮喪我很快便忘記不愉快的事情面對困難時 , 我仍能保持樂觀

18/2/2002 J.Y.Y. Kwong 15

Validity

Criterion-related validity: IQ test score relates to academic performance, speed of learning new wordsStandford Binet IQ test: Hi score more ‘A’ grades than low score占卜 ( 有橫財 ) – 買六合彩

There is evidence for CR validity when the test score relates to another variable which is theoretically related (objective).

18/2/2002 J.Y.Y. Kwong 16

Validity

Criterion-related validity: ExamplesO-P

Hi score in O-P less likely to be depressed; low score more likely to be

Psychological assessment scores should relate to job performance!

Higher Ability tests score – higher performance rating

18/2/2002 J.Y.Y. Kwong 17

A good test has evidence for its validity:

Face – the test “looks like” what it intends to measure (Picture drawing – interpersonal skills)

Content – the content of the test has adequate

coverage of the variable ( 整齊清潔 = 好男友？ )

Criterion-related – test score relates to another theoretically related variable (IQ – “A”;

Ability test score – job performance)

18/2/2002 J.Y.Y. Kwong 18

Typical Criterion-related Validity CoefficientsAge r = .00Handwriting analysis (Graphology) .00Vocational interests tests .10College grades .13Reference checks .17Experience .18Personality tests .24Assessment centers .25Structured interviews .34Biographical information blanks .35Job knowledge tests .50Cognitive and psychomotor ability tests.53

18/2/2002 J.Y.Y. Kwong 19

How to pick a good assessment test?

Ask for information about the test’sReliabilityValidity

JA Task & KSAOs Pick some reliable and valid assessment tests Now, we know how to assess the job applicants!NEXT, recruitment and selection