V Seminario TEST DE JUICIO SITUACIONALcatedras.iic.uam.es/Publicaciones/V_seminario_MAP.pdf · 2-3 • Programación 5 • Prof. Filip Lievens 6-29 • Prof. Fiona Patterson 30-41

V SeminarioTEST DE JUICIO SITUACIONAL

• introducción

2-3

• Programación

5

• Prof. Filip Lievens

6-29

• Prof. Fiona Patterson

30-41

• mr. Stefan meyer

42-49

• Prof. Julio olea

50-59

• resumen posters

60-67

• notas

4—5

4—5

el objetivo de los seminarios de la Cátedra

maP es fomentar y contribuir a la medición

psicológica de calidad, especialmente en

el campo de las organizaciones.

en este quinto seminario se ha escogido

como tema central los test situacionales.

•

26 de junio de 2013

Facultad de Psicología

Universidad autónoma de madrid

6—7

PROGRAMA

6—7

Recogida del material del seminario.

Apertura y presentación.Ángela LoechesDecana de la facultad de Psicología de la Uam.

José miguel mataDirector General del instituto de ingeniería del

Conocimiento.

Conferencia. Situational Judgment Tests: an introduction to theory, practice, and research.Prof. Filip LievensProfesor del departamento de Gestión de personal y

Psicología del trabajo y de las organizaciones de la

Universidad de Gante de Bélgica.

Sesión de pósters. Café.

Conferencia. evaluations of situational judgments tests for high stakes selection.Prof. Fiona PattersonProfesora del departamento de Psicología de la

Universidad de Cambridge y directora del Work

Psychology Group.

Conferencia. Situational Judgment Test in ePSo open competitions for the european institutions from 2010 to present.mr. Stefan meyerresponsable de evaluaciones informatizadas de la

european Personnel Selection office (ePSo).

Conferencia. medición de competencias mediante modelos de Diagnóstico Cognitivo: aplicación a un Test de Juicio Situacional.Prof. Julio oleaCatedrático de Psicología de la Universidad autónoma

de madrid y codirector de la cátedra Uam-iiC modelos y

aplicaciones psicométricos (maP).

Debate y clausura del seminario.

8:45

9:15

9:30

10:40

11:40

12:20

13:00

13:40

8—98—9

Situational Judgment Tests: An introduction to theory, practice, and research.— Prof. Filip Lievens

Departamento de Gestión de Personal y Psicología del Trabajo y de las organizaciones, de la Universidad de Gante, Bélgica.

8—98—9

1

Situational Judgment Tests: An Introduction to Theory,

Practice, & Research

Madrid June, 2013

Prof. Dr. Filip Lievens Ghent University, Belgium

Department of Personnel Dunantlaan 2, 9000, Ghent [email protected]

Management and Work and Organizational Psychology http://users.ugent.be /~flievens/

Who am I? !  Professor: Ghent U. (Belgium)

!  Visiting professor: !  U. of Minnesota (USA) !  Bowling Green State U. (USA) !  U. of Zürich (Switzerland) !  U. of Guelph (Canada) !  Nanyang Technological U. (Singapore) !  Singapore Management U. (Singapore) !  U. of Valencia (Spain) !  U. of Giessen (Germany) !  U. of Stellenbosch (South Africa)

1

Situational Judgment Tests: An Introduction to Theory,

Practice, & Research

Madrid June, 2013

Prof. Dr. Filip Lievens Ghent University, Belgium

Department of Personnel Dunantlaan 2, 9000, Ghent [email protected]

Management and Work and Organizational Psychology http://users.ugent.be /~flievens/

Who am I? !  Professor: Ghent U. (Belgium)

!  Visiting professor: !  U. of Minnesota (USA) !  Bowling Green State U. (USA) !  U. of Zürich (Switzerland) !  U. of Guelph (Canada) !  Nanyang Technological U. (Singapore) !  Singapore Management U. (Singapore) !  U. of Valencia (Spain) !  U. of Giessen (Germany) !  U. of Stellenbosch (South Africa)

2

Who am I? !  Research expertise

!  Situational judgment tests, high-stakes testing, assessment centers, web-based assessment, & employer branding

!  Teaching at Ghent U. (Belgium) !  HRM

!  Consultancy !  Private, public, & military sector !  Metaconsultant of international consultancy firms

3

!  Theory !  Basics of SJTs !  History !  Definition

!  Practice !  SJT development & SJT building bocks

!  State-of-the-art of SJT research

Objectives & overview

4

10—11

2

Who am I? !  Research expertise

!  Situational judgment tests, high-stakes testing, assessment centers, web-based assessment, & employer branding

!  Teaching at Ghent U. (Belgium) !  HRM

!  Consultancy !  Private, public, & military sector !  Metaconsultant of international consultancy firms

3

!  Theory !  Basics of SJTs !  History !  Definition

!  Practice !  SJT development & SJT building bocks

!  State-of-the-art of SJT research

Objectives & overview

4

3

Roles

SJT developer or SJT user

SJT Candidate

Trainer /supervisor of SJT

users

SJT researcher

5

SJT Theory: Basics of SJTs

3

Roles

SJT developer or SJT user

SJT Candidate

Trainer /supervisor of SJT

users

SJT researcher

5

SJT Theory: Basics of SJTs

Situational Judgment Tests: an introduction to theory, practice, and research.

10—11

4

Suppose you have to assess people on their level of customer service during selection? What do you do? !  Personality inventory

!  Agreeableness, Emotional Stability, & Conscientiousness

!  Interview

!  Role-play (high-fidelity simulation)

!  SJT (low-fidelity simulation)

Sign-based/dispositional

approach

Sample-based/interactional

approach

SJT Basics Written SJT item

SJT Introduction Prof. Dr. Filip Lievens

You have an excellent long term employee who is always visiting at length with customers. Although customer orientation is one of the key values in your company, other employees have complained that the employee is not doing her share of the administrative work. Pick the best response.

•  Leave her alone since the customers seem to enjoy the chats.

•  Tell her you need her to work more and talk less and then give her assignments to be completed by specific times.

•  Initiate formal discipline against her.

•  Explain the difference between social chatter and service and coach her on how to keep her conversations shorter.

•  Discuss the issues with the employee only if her production is below standard.


8

4

Suppose you have to assess people on their level of customer service during selection? What do you do? !  Personality inventory

!  Agreeableness, Emotional Stability, & Conscientiousness

!  Interview

!  Role-play (high-fidelity simulation)

!  SJT (low-fidelity simulation)

Sign-based/dispositional

approach

Sample-based/interactional

approach




•  Leave her alone since the customers seem to enjoy the chats.

•  Tell her you need her to work more and talk less and then give her assignments to be completed by specific times.

•  Initiate formal discipline against her.

•  Explain the difference between social chatter and service and coach her on how to keep her conversations shorter.

•  Discuss the issues with the employee only if her production is below standard.


8

5

You work as a waiter/waitress and a customer orders a dish that is not on the menu. You mention this politely but the customer gets very upset. What do you do?

!  Suggest alternative main dishes and side orders that might be of interest

!  Suggest alternative dishes that might be of interest to the customer

!  Suggest only those alternative dishes that are very similar to the dish the customer wanted

!  Tell the customer there is nothing you can do

9


SJT Basics What Are SJTs?

!  An applicant is presented with a situation & asked what he/she would do.

!  SJT items are typically in a multiple choice format: items have a stem and various item responses (response options).

SJT Introduction Prof. Dr. Filip Lievens 10

12—13

5

You work as a waiter/waitress and a customer orders a dish that is not on the menu. You mention this politely but the customer gets very upset. What do you do?

!  Suggest alternative main dishes and side orders that might be of interest

!  Suggest alternative dishes that might be of interest to the customer

!  Suggest only those alternative dishes that are very similar to the dish the customer wanted

!  Tell the customer there is nothing you can do

9


SJT Basics What Are SJTs?

!  An applicant is presented with a situation & asked what he/she would do.

!  SJT items are typically in a multiple choice format: items have a stem and various item responses (response options).


6


SJT Basics History of SJTs

!  1873: US Civil Service Examinations !  « A banking company asks protection for a certain

device, as a trade-mark, which they propose to put upon their notes. What action would you take in the application? »

!  1905: Binet !  « When a person has offended you and comes to offer

his apologies, what would you do? » !  1941: Ansbacher

!  Your sports club is planning a trip to Berlin to attend the Germany-England football game, which will take place in 2 weeks. You have been entrusted with the preparations and entire management of the trip. What do you intend to do? 11



!  Early years !  1926: Judgment scale in the George Washington

University Social Intelligence Test !  Used in World War II !  1948: How Supervise? (Rosen, 1961) !  1960’s: SJTs used at the U.S. Civil Service System

!  Breakthrough years !  Motowidlo reinvigorated interest: low fidelity

simulations . !  Sternberg reinvigorated interest: tacit knowledge

inventories . 12

6



!  1873: US Civil Service Examinations !  « A banking company asks protection for a certain

device, as a trade-mark, which they propose to put upon their notes. What action would you take in the application? »

!  1905: Binet !  « When a person has offended you and comes to offer

his apologies, what would you do? » !  1941: Ansbacher

!  Your sports club is planning a trip to Berlin to attend the Germany-England football game, which will take place in 2 weeks. You have been entrusted with the preparations and entire management of the trip. What do you intend to do? 11



!  Early years !  1926: Judgment scale in the George Washington

University Social Intelligence Test !  Used in World War II !  1948: How Supervise? (Rosen, 1961) !  1960’s: SJTs used at the U.S. Civil Service System

!  Breakthrough years !  Motowidlo reinvigorated interest: low fidelity

simulations . !  Sternberg reinvigorated interest: tacit knowledge

inventories . 12


12—13

7


SJT Basics In 2013

!  Popular in US & UK !  Public & private sector !  EPSO

!  Emerging body of research !  2006: first SJT book (edited by

Weekley & Ployhart)

!  Increasing demand around the world

14

7


SJT Basics In 2013

!  Popular in US & UK !  Public & private sector !  EPSO

!  Emerging body of research !  2006: first SJT book (edited by

Weekley & Ployhart)

!  Increasing demand around the world

14

8

SJT Basics Research on SJTs

0

10

20

30

40

50

60

<91 91-95 96-00 01-05 06-10

Freq

uenc

y

Years

SJT Publications in Web of Science


SJT Basics In 2013

16

14—15

8

SJT Basics Research on SJTs

0

10

20

30

40

50

60

<91 91-95 96-00 01-05 06-10

Freq

uenc

y

Years

SJT Publications in Web of Science


SJT Basics In 2013

16

9


SJT Basics Wat do SJTs measure?

!  Measurement method that can be designed to measure a variety of constructs

!  Procedural knowledge about costs & benefits of courses of action in a particular domain !  Going beyond cognitive ability !  Mostly interpersonal & leadership competencies

17

SJT Basics Theory of Prediction

Situation Person

Behavior

Performance

On-the-job

S

election

Performance

Behavior

Knowledge

Situation Person

Job specific

General

IQ

Personality

Socialization

Experience

9


SJT Basics Wat do SJTs measure?

!  Measurement method that can be designed to measure a variety of constructs

!  Procedural knowledge about costs & benefits of courses of action in a particular domain !  Going beyond cognitive ability !  Mostly interpersonal & leadership competencies

17

SJT Basics Theory of Prediction

Situation Person

Behavior

Performance

On-the-job

S

election

Performance

Behavior

Knowledge

Situation Person

Job specific

General

IQ

Personality

Socialization

Experience


14—15

10


SJT Basics Comparisons with tests

!  Parallels !  Standardization !  Automatic scoring !  Performance test !  Large group screening

!  Selecting-out

!  Differences !  Sample-based vs. sign-based !  Contextualized vs. decontextualized !  Multidimensional vs. unidimensional !  Procedural knowledge vs. aptitude & declarative knowledge !  Noncognitive competencies vs. cognitive competencies

19



SJT Basics Comparisons with assessment centers

!  Parallels !  Behavioral consistency logic !  Sample-based (vs. sign-based) !  Psychological fidelity !  Multidimensional: Methods which measure multiple constructs

!  Differences !  Stimulus

!  Standardized situation vs. life situation !  Low fidelity vs. high fidelity

!  Response !  Procedural knowledge vs. actual behavior !  MC vs. open ended

!  Scoring !  A priori vs. life observation/rating !  Built-in model vs. trained assessors

!  Use !  Large vs. small groups !  Select out vs. select in 20

10


SJT Basics Comparisons with tests

!  Parallels !  Standardization !  Automatic scoring !  Performance test !  Large group screening

!  Selecting-out

!  Differences !  Sample-based vs. sign-based !  Contextualized vs. decontextualized !  Multidimensional vs. unidimensional !  Procedural knowledge vs. aptitude & declarative knowledge !  Noncognitive competencies vs. cognitive competencies

19



SJT Basics Comparisons with assessment centers

!  Parallels !  Behavioral consistency logic !  Sample-based (vs. sign-based) !  Psychological fidelity !  Multidimensional: Methods which measure multiple constructs


!  Standardized situation vs. life situation !  Low fidelity vs. high fidelity

!  Response !  Procedural knowledge vs. actual behavior !  MC vs. open ended

!  Scoring !  A priori vs. life observation/rating !  Built-in model vs. trained assessors

!  Use !  Large vs. small groups !  Select out vs. select in 20

11

SJT Basics Comparisons with behavior description interviews

!  Parallels !  Behavioral consistency logic !  Sample-based (vs. sign-based) !  Multidimensional: Methods which measure multiple constructs


!  Standardized situation vs. situation reported by interviewee !  Past or future situation vs. past situation

!  Response !  Written response vs. oral response !  MC vs. open ended

!  Scoring !  A priori vs. life observation & rating !  Built-in model vs. trained interviewers

!  Use !  Large vs. small groups


21

SJT Practice:

SJT Development

16—17

11

SJT Basics Comparisons with behavior description interviews

!  Parallels !  Behavioral consistency logic !  Sample-based (vs. sign-based) !  Multidimensional: Methods which measure multiple constructs


!  Standardized situation vs. situation reported by interviewee !  Past or future situation vs. past situation

!  Response !  Written response vs. oral response !  MC vs. open ended

!  Scoring !  A priori vs. life observation & rating !  Built-in model vs. trained interviewers

!  Use !  Large vs. small groups


21

SJT Practice:

SJT Development

12


SJT Development SJT vendors

!  General !  SHL / Previsor !  A & DC !  Kronos !  Aon !  …

!  Specific !  Work Skills First !  Ergometrics !  Van der Maesen/Koch

23

SJT Development Stages in SJT Development !  Strategic SJT decisions

!  Determine purpose of SJT !  Identify target population

!  Content development of SJT !  Generate critical incidents !  Sort critical incidents !  Turn selected critical incidents into item stems !  Generate item responses !  Edit item responses

!  Instructions & scoring !  Determine response instructions !  Develop a scoring key

!  Implementation !  Continued use of SJTs


12


SJT Development SJT vendors

!  General !  SHL / Previsor !  A & DC !  Kronos !  Aon !  …

!  Specific !  Work Skills First !  Ergometrics !  Van der Maesen/Koch

23

SJT Development Stages in SJT Development !  Strategic SJT decisions

!  Determine purpose of SJT !  Identify target population

!  Content development of SJT !  Generate critical incidents !  Sort critical incidents !  Turn selected critical incidents into item stems !  Generate item responses !  Edit item responses

!  Instructions & scoring !  Determine response instructions !  Develop a scoring key

!  Implementation !  Continued use of SJTs



16—17

13

SJT Development SJT Building Blocks

25

1.  Item length 2.  Item complexity 3.  Item comprehensibility 4.  Item contextualization 5.  Item branching 6.  Response & stimulus fidelity 7.  Response instructions 8.  Scoring key 9.  Measurement level of scores 10. Constructs measure

25

Pur

pose


SJT Development SJT Purpose

!  Selection & assessment

!  Recruitment (realistic job preview)

!  Training & evaluation (scenario-based)

!  Promotion

26

13

SJT Development SJT Building Blocks

25

1.  Item length 2.  Item complexity 3.  Item comprehensibility 4.  Item contextualization 5.  Item branching 6.  Response & stimulus fidelity 7.  Response instructions 8.  Scoring key 9.  Measurement level of scores 10. Constructs measure

25

Pur

pose


SJT Development SJT Purpose

!  Selection & assessment

!  Recruitment (realistic job preview)

!  Training & evaluation (scenario-based)

!  Promotion

26

14

SJT Development 1. Item Length

!  Stems vary in the length of the situation presented. !  Very short descriptions of situation

!  E.g., Your colleague is rude towards you .

!  Very detailed descriptions of situation !  E.g., Tacit Knowledge Inventory (Wagner & Sternberg,

1991)


SJT Development

Tacit Knowledge Inventory (Sternberg)

You are a company commander, and your battalion commander is the type of person who seems always to shoot the messenger – he does not like to be surprised by bad news, and he tends to take his anger out on the person who brought him the bad news. You want to build a positive, professional relationship with your battalion commander. What should you do?

___ Speak to your battalion commander about his behavior and share your perception of it. ___ Attempt to keep the battalion commander over-informed by telling him what is

occurring in your unit on a regular basis (e.g., daily or every other day). ___ Speak to the sergeant major and see if she/he is willing to try to influence the battalion

commander. ___ Keep the battalion commander informed only on important issues, but don t bring up

issues you don t have to discuss with him. ___ When you bring a problem to your battalion commander, bring a solution at the same

time. ___ Disregard the battalion commander s behavior: Continue to bring him news as you

normally would. ___ Tell your battalion commander all of the good news you can, but try to shield him from

hearing the bad news. ___ Tell the battalion commander as little as possible; deal with problems on your own if at

all possible.

28

18—19

14

SJT Development 1. Item Length

!  Stems vary in the length of the situation presented. !  Very short descriptions of situation

!  E.g., Your colleague is rude towards you .

!  Very detailed descriptions of situation !  E.g., Tacit Knowledge Inventory (Wagner & Sternberg,

1991)


SJT Development

Tacit Knowledge Inventory (Sternberg)

You are a company commander, and your battalion commander is the type of person who seems always to shoot the messenger – he does not like to be surprised by bad news, and he tends to take his anger out on the person who brought him the bad news. You want to build a positive, professional relationship with your battalion commander. What should you do?

___ Speak to your battalion commander about his behavior and share your perception of it. ___ Attempt to keep the battalion commander over-informed by telling him what is

occurring in your unit on a regular basis (e.g., daily or every other day). ___ Speak to the sergeant major and see if she/he is willing to try to influence the battalion

commander. ___ Keep the battalion commander informed only on important issues, but don t bring up

issues you don t have to discuss with him. ___ When you bring a problem to your battalion commander, bring a solution at the same

time. ___ Disregard the battalion commander s behavior: Continue to bring him news as you

normally would. ___ Tell your battalion commander all of the good news you can, but try to shield him from

hearing the bad news. ___ Tell the battalion commander as little as possible; deal with problems on your own if at

all possible.

28

15


SJT Development 2. Item Complexity

!  Stems vary in the complexity of the situation presented. !  Low complexity

!  One has difficulty with a new assignment and needs instructions .

!  High complexity !  One has multiple supervisors who are not

cooperating with each other, and who are providing conflicting instructions concerning which of your assignments has highest priority .

29

SJT Development 3. Item Comprehensiblity

!  Comprehensibility: It is more difficult to understand the meaning and import of some situations than other situations !  cf. reading level formulas, technical jargon

!  Length, complexity, & comprehensibility of the situations are interrelated and probably drive the cognitive loading of items.


15


SJT Development 2. Item Complexity

!  Stems vary in the complexity of the situation presented. !  Low complexity

!  One has difficulty with a new assignment and needs instructions .

!  High complexity !  One has multiple supervisors who are not

cooperating with each other, and who are providing conflicting instructions concerning which of your assignments has highest priority .

29

SJT Development 3. Item Comprehensiblity

!  Comprehensibility: It is more difficult to understand the meaning and import of some situations than other situations !  cf. reading level formulas, technical jargon

!  Length, complexity, & comprehensibility of the situations are interrelated and probably drive the cognitive loading of items.



18—19

16

Patient : So, this physiotherapy is really going to help me?

Physician: Absolutely, even though the first days it might still be painful.

Patient : Yes, I suppose it will take a while before it starts working.

Physician: That is why I am going to prescribe a painkiller. You should take 3 painkillers per day.

Patient: Do I really have to take them? I have already tried a few things. First, they didn t help me. And second, I m actually opposed to taking any medication. I d rather not take them. They are not good for my health.

What is the best way for you (as a physician) to react to this patient s refusal to take the prescribed medication?

a. Ask her if she knows something else to relieve the pain.

b. Give her the scientific evidence as to why painkillers will help.

c. Agree not to take them now but also stress the importance of the physiotherapy.

d. Tell her that, in her own interest, she will have to start changing her attitude. 31

SJT Development Example admission SJT item

A 55 year old woman with ischaemic heart disease has smoked 20 cigarettes per day for 40 years. She requests nicotine replacement patches. She has had these previously but has been inconsistent in their use and has often continued to smoke while using the patches.

A. Emphasize the dangers of smoking but do not prescribe.

B. Enquire about the difficulties she has with stopping smoking and any previous problems with patches.

C. Insist on a period of abstinence before prescribing any further patches.

D. Prescribe another supply of patches and explain how they should be used.

E. Suggest that nicotine replacement therapy is not suitable for her but explore alternative therapies.

32

SJT Development Example advanced level SJT item

16

Patient : So, this physiotherapy is really going to help me?

Physician: Absolutely, even though the first days it might still be painful.

Patient : Yes, I suppose it will take a while before it starts working.

Physician: That is why I am going to prescribe a painkiller. You should take 3 painkillers per day.

Patient: Do I really have to take them? I have already tried a few things. First, they didn t help me. And second, I m actually opposed to taking any medication. I d rather not take them. They are not good for my health.

What is the best way for you (as a physician) to react to this patient s refusal to take the prescribed medication?

a. Ask her if she knows something else to relieve the pain.

b. Give her the scientific evidence as to why painkillers will help.

c. Agree not to take them now but also stress the importance of the physiotherapy.

d. Tell her that, in her own interest, she will have to start changing her attitude. 31

SJT Development Example admission SJT item

A 55 year old woman with ischaemic heart disease has smoked 20 cigarettes per day for 40 years. She requests nicotine replacement patches. She has had these previously but has been inconsistent in their use and has often continued to smoke while using the patches.

A. Emphasize the dangers of smoking but do not prescribe.

B. Enquire about the difficulties she has with stopping smoking and any previous problems with patches.

C. Insist on a period of abstinence before prescribing any further patches.

D. Prescribe another supply of patches and explain how they should be used.

E. Suggest that nicotine replacement therapy is not suitable for her but explore alternative therapies.

32

SJT Development Example advanced level SJT item

17

SJT Development 4. Item Contextualization

!  Items for a job can be more specific. !  Mention job specific equipment, technical terms.

!  Items for a family of jobs need to make sense for all the jobs to be covered by the test. !  Entry-level vs. Supervisory !  Armed Forces, Coast Guard, Paramilitary Forces, &

Security Forces

!  Items for specific competencies needed across several jobs or job families to be covered by SJT. !  SJT taxonomy


Judgment at Work Survey for Customer Service © Work Skills First, Inc.

!  A customer is disrespectful and rude to you. !  Ask the customer to leave until he is calm. !  Treat the customer as the customer treats you. !  Try to solve the problem as quickly as possible to get the customer out of the store.

!  A customer is very angry with you because the customer believes that the product received is not what the customer !  Offer to replace the customer's product with the correct one. !  Ask a co-worker to deal with this problem.

!  A customer is becoming impatient about waiting for service for a long time. Other customers are becoming equally anxious. !  Apologize for the slow service on the part of your colleagues. !  Offer the customers a discount. !  Reassure the customer that the wait is necessary and address the needs they may

have. !  Try to find someone else to wait on the customer immediately. !  Refer the customer to a competing organization that offers similar services. !  Suggest that the customer consider leaving and coming back when you are not so

busy.

Extremely Ineffective - Ineffective -Average Effectiveness – Effective -Extremely Effective 34

20—21

17

SJT Development 4. Item Contextualization

!  Items for a job can be more specific. !  Mention job specific equipment, technical terms.

!  Items for a family of jobs need to make sense for all the jobs to be covered by the test. !  Entry-level vs. Supervisory !  Armed Forces, Coast Guard, Paramilitary Forces, &

Security Forces

!  Items for specific competencies needed across several jobs or job families to be covered by SJT. !  SJT taxonomy


Judgment at Work Survey for Customer Service © Work Skills First, Inc.

!  A customer is disrespectful and rude to you. !  Ask the customer to leave until he is calm. !  Treat the customer as the customer treats you. !  Try to solve the problem as quickly as possible to get the customer out of the store.

!  A customer is very angry with you because the customer believes that the product received is not what the customer !  Offer to replace the customer's product with the correct one. !  Ask a co-worker to deal with this problem.

!  A customer is becoming impatient about waiting for service for a long time. Other customers are becoming equally anxious. !  Apologize for the slow service on the part of your colleagues. !  Offer the customers a discount. !  Reassure the customer that the wait is necessary and address the needs they may

have. !  Try to find someone else to wait on the customer immediately. !  Refer the customer to a competing organization that offers similar services. !  Suggest that the customer consider leaving and coming back when you are not so

busy.

Extremely Ineffective - Ineffective -Average Effectiveness – Effective -Extremely Effective 34

18

SJT Development 5. Item Branching / Integration

!  Unbranched/linear items

!  Branched/nonlinear items !  Present overall situation followed by subordinate

situations. !  Subordinate stems are stems linked to the responses.



SJT Development 6. Stimulus & Response Fidelity

!  Degree to which format of stimulus (item stem) and response is consistent with how situation is encountered in a work setting.

!  Stimulus !  Written !  Oral !  Video !  Behavior

!  Response !  Written !  Oral !  Video !  Behavior

Written SJT

Situational interview

Assessment center

Webcam SJT Video-based SJT

36

18

SJT Development 5. Item Branching / Integration

!  Unbranched/linear items

!  Branched/nonlinear items !  Present overall situation followed by subordinate

situations. !  Subordinate stems are stems linked to the responses.



SJT Development 6. Stimulus & Response Fidelity

!  Degree to which format of stimulus (item stem) and response is consistent with how situation is encountered in a work setting.

!  Stimulus !  Written !  Oral !  Video !  Behavior

!  Response !  Written !  Oral !  Video !  Behavior

Written SJT


Assessment center

Webcam SJT Video-based SJT

36


20—21

19


Video-based SJT

Multimedia SJT

Level of fidelity

Level of standardization

Role-play

Group discussion

Webcam SJT

Written SJT


Cartoon SJT

37


A.  Leave her alone since the customers seem to enjoy the chats.

B.  Tell her you need her to work more and talk less and then give her assignments to be completed by specific times.

C.  Initiate formal discipline against her.

D.  Explain the difference between social chatter and service and coach her on how to keep her conversations shorter.

E.  Discuss the issues with the employee only if her production is below standard.

Written SJT

Video SJT

3D animated SJT

Avatar-based SJT

© Previsor

SJT Development SJT generations

Cartoon SJT

Webcam SJT

38

19


Video-based SJT

Multimedia SJT

Level of fidelity

Level of standardization

Role-play

Group discussion

Webcam SJT

Written SJT


Cartoon SJT

37


A.  Leave her alone since the customers seem to enjoy the chats.

B.  Tell her you need her to work more and talk less and then give her assignments to be completed by specific times.

C.  Initiate formal discipline against her.

D.  Explain the difference between social chatter and service and coach her on how to keep her conversations shorter.

E.  Discuss the issues with the employee only if her production is below standard.

Written SJT

Video SJT

3D animated SJT

Avatar-based SJT

© Previsor

SJT Development SJT generations

Cartoon SJT

Webcam SJT

38

20

SJT Development 7. Response Instructions

39

!  Knowledge instructions ask respondents to display their knowledge of the effectiveness of behavioral responses !  Best action

!  Pick the best answer. !  Best action/worst action

!  Pick the best & worst answer.

!  Rate on effectiveness !  Rate each response for

effectiveness.

!  Behavioral tendency instructions ask respondents to report how they typically respond !  Most likely action

!  What would you most likely do? !  Most likely action/least likely

action !  What would you most likely do?

What would you least likely do? !  Rate on the likelihood of

performing the action !  Rate each response on

likelihood that you would do the behavior.


SJT Development

7. Response instructions Knowledge Behavioral consistency

Cognitive loading

More correlated with cognitive ability (.35)

Less correlated with cognitive ability (.19)

Personality loading

Less correlated with A (.19), C (.24), ES (.12)

More correlated with A (.37), C (.34), ES (.35)

Mean SJT scores

Higher Lower

Race differences

Larger Smaller

Fakability Less More

Validity ρ = .26 ρ = .26 40

22—23

20

SJT Development 7. Response Instructions

39

!  Knowledge instructions ask respondents to display their knowledge of the effectiveness of behavioral responses !  Best action

!  Pick the best answer. !  Best action/worst action

!  Pick the best & worst answer.

!  Rate on effectiveness !  Rate each response for

effectiveness.

!  Behavioral tendency instructions ask respondents to report how they typically respond !  Most likely action

!  What would you most likely do? !  Most likely action/least likely

action !  What would you most likely do?

What would you least likely do? !  Rate on the likelihood of

performing the action !  Rate each response on

likelihood that you would do the behavior.


SJT Development

7. Response instructions Knowledge Behavioral consistency

Cognitive loading

More correlated with cognitive ability (.35)

Less correlated with cognitive ability (.19)

Personality loading

Less correlated with A (.19), C (.24), ES (.12)

More correlated with A (.37), C (.34), ES (.35)

Mean SJT scores

Higher Lower

Race differences

Larger Smaller

Fakability Less More

Validity ρ = .26 ρ = .26 40

21

SJT Development 8. Scoring Key

!  Rational (consensus among experts) !  E.g., 60% agreement

!  Empirical (administration of large groups) !  E.g., 25% endorsement of correct option

!  Theoretical model !  E.g., conflict management, leadership

!  Hybrid !  Combination of approaches 41


!  Most use discrete points assigned to response options. !  From a legal point of view, it makes some uneasy to try to

defend the difference between very effective and effective. !  Very effective = 1 !  Effective = 1 !  Ineffective = 0 !  Very ineffective = 0

!  Others use score responses as deviations from the mean effectiveness ratings (= the correct answer) !  If the mean is 1.5, a respondent who provided a rating of 1 or

2 would both have a -.5 as a score on the item. !  Zero is the highest possible score !  Prone to large coaching effects (Cullen et al., 2006)

42


21


!  Rational (consensus among experts) !  E.g., 60% agreement

!  Empirical (administration of large groups) !  E.g., 25% endorsement of correct option

!  Theoretical model !  E.g., conflict management, leadership

!  Hybrid !  Combination of approaches 41


!  Most use discrete points assigned to response options. !  From a legal point of view, it makes some uneasy to try to

defend the difference between very effective and effective. !  Very effective = 1 !  Effective = 1 !  Ineffective = 0 !  Very ineffective = 0

!  Others use score responses as deviations from the mean effectiveness ratings (= the correct answer) !  If the mean is 1.5, a respondent who provided a rating of 1 or

2 would both have a -.5 as a score on the item. !  Zero is the highest possible score !  Prone to large coaching effects (Cullen et al., 2006)

42



22—23

22

SJT Development 9. Measurement Level of Scores

43

!  One dichotomous response per item !  Most likely, pick the best

!  Two dichotomous responses per item !  Most/least likely, pick the best/worst

!  Ranking the responses yields ordinal level data.

!  Rating the effectiveness yields interval level data per item response.

SJT Development 10. Constructs measured

!  Homogeneous !  One construct

!  Heterogeneous !  Multiple constructs


Construct Usage

Technical knowledge 2.94%

Interpersonal Skills 12.50%

Teamwork Skills 4.41%

Leadership 37.50%

Basic personality 9.56%

Heterogeneous 33.09%

Christian et al., 2010; N= 136 SJTs

22

SJT Development 9. Measurement Level of Scores

43

!  One dichotomous response per item !  Most likely, pick the best

!  Two dichotomous responses per item !  Most/least likely, pick the best/worst

!  Ranking the responses yields ordinal level data.

!  Rating the effectiveness yields interval level data per item response.

SJT Development 10. Constructs measured

!  Homogeneous !  One construct

!  Heterogeneous !  Multiple constructs


Construct Usage

Technical knowledge 2.94%

Interpersonal Skills 12.50%

Teamwork Skills 4.41%

Leadership 37.50%

Basic personality 9.56%

Heterogeneous 33.09%

Christian et al., 2010; N= 136 SJTs

23

SJT Research: State-of-the-art


Research

SJT Quiz

Indica si la siguiente afirmación es “verdadera” o “falsa”:

1.  La validez predictiva de los tests de juicio situacional (SJT) es aproximadamente 0.3.

2.  Los SJTs mejoran la predicción del rendimiento que puede alcanzarse con las pruebas de capacidad

cognitiva y personalidad.

3.  Los SJTs basados en video son mejores predictores del rendimiento futuro que los SJTs en papel.

4.  Los hombres puntúan por lo general más alto que las mujeres en los SJTs.

5.  Se han encontrado diferencias raciales de escasa importancia en los SJTs.

6.  Las diferencias raciales en los SJTs son mas pequeñas en los SJTs basados en video.

7.  No se han encontrado diferencias entre la administración informatizada de los SJTs y la tradicional

(administración en papel).

8.  Cuando se realiza un análisis factorial sobre las puntuaciones factoriales, se suele encontrar una

estructura factorial interpretable.

9.  La fiabilidad test-retest de los SJT es aproximadamente 0.70.

10.  Es posible entrenar a la gente para que mejore su rendimiento en los SJTs.

24—25

23

SJT Research: State-of-the-art


Research

SJT Quiz

Indica si la siguiente afirmación es “verdadera” o “falsa”:

1.  La validez predictiva de los tests de juicio situacional (SJT) es aproximadamente 0.3.

2.  Los SJTs mejoran la predicción del rendimiento que puede alcanzarse con las pruebas de capacidad

cognitiva y personalidad.

3.  Los SJTs basados en video son mejores predictores del rendimiento futuro que los SJTs en papel.

4.  Los hombres puntúan por lo general más alto que las mujeres en los SJTs.

5.  Se han encontrado diferencias raciales de escasa importancia en los SJTs.

6.  Las diferencias raciales en los SJTs son mas pequeñas en los SJTs basados en video.

7.  No se han encontrado diferencias entre la administración informatizada de los SJTs y la tradicional

(administración en papel).

8.  Cuando se realiza un análisis factorial sobre las puntuaciones factoriales, se suele encontrar una

estructura factorial interpretable.

9.  La fiabilidad test-retest de los SJT es aproximadamente 0.70.

10.  Es posible entrenar a la gente para que mejore su rendimiento en los SJTs.

24


Research

Internal consistency

!  Meta-analysis (Catano, Brochu & Lamerson, 2012)

!  Average alpha = .46 (N=56) !  Factor analysis: Several factors & no clear structure

!  Moderators (McDaniel et al., 2001)

!  Length of SJT !  Type of scale

!  Rating scale (.73) > Two choices (.60) > One choice (.24)

48

Research

Test-retest Reliability

!  Adequate !  Between .60 - .80

24


Research

Internal consistency

!  Meta-analysis (Catano, Brochu & Lamerson, 2012)

!  Average alpha = .46 (N=56) !  Factor analysis: Several factors & no clear structure

!  Moderators (McDaniel et al., 2001)

!  Length of SJT !  Type of scale

!  Rating scale (.73) > Two choices (.60) > One choice (.24)

48

Research

Test-retest Reliability

!  Adequate !  Between .60 - .80


24—25

25

Research

Criterion-related Validity !  McDaniel, Morgeson, & Campion (2001)

!  ρ = .27 (corrected r = .34)

!  McDaniel et al. (2007) !  ρ = .20 (corrected r = .26)

!  Validity for predicting relevant criteria (Christian et al., 2010)

!  Equivalence computer & paper-and-pencil SJTs (Ployhart et al., 2003)

!  Moderators: McDaniel, Morgeson & Campion (2001) !  Job analysis (.38 vs. .29) !  Less detailed questions (.35 vs. 33) !  Video-based (Christian et al., 2010)

Research

Incremental Validity (McDaniel et al., 2007)

!  Incremental validity over !  Cognitive ability (3%-5%) !  Personality (6%-7%) !  Cognitive ability & personality (1%-2%)

50

25

Research

Criterion-related Validity !  McDaniel, Morgeson, & Campion (2001)

!  ρ = .27 (corrected r = .34)

!  McDaniel et al. (2007) !  ρ = .20 (corrected r = .26)

!  Validity for predicting relevant criteria (Christian et al., 2010)

!  Equivalence computer & paper-and-pencil SJTs (Ployhart et al., 2003)

!  Moderators: McDaniel, Morgeson & Campion (2001) !  Job analysis (.38 vs. .29) !  Less detailed questions (.35 vs. 33) !  Video-based (Christian et al., 2010)

Research

Incremental Validity (McDaniel et al., 2007)

!  Incremental validity over !  Cognitive ability (3%-5%) !  Personality (6%-7%) !  Cognitive ability & personality (1%-2%)

50

26

Note. Correlations corrected for range restriction and unreliability in criterion.

-0,10 0,00 0,10 0,20 0,30 0,40 0,50 0,60

Cognitive Tests

Video-based Interpersonal SJT

52

Research

Gender Differences

!  Females > males (d = .10)

!  Explanation !  Not related to cognitive loading & response instructions

!  Interpersonal nature of situations? !  Cf. assessment center/work sample research


26—27

26

Note. Correlations corrected for range restriction and unreliability in criterion.

-0,10 0,00 0,10 0,20 0,30 0,40 0,50 0,60

Cognitive Tests

Video-based Interpersonal SJT

52

Research

Gender Differences

!  Females > males (d = .10)

!  Explanation !  Not related to cognitive loading & response instructions

!  Interpersonal nature of situations? !  Cf. assessment center/work sample research


27


Research

Race differences

!  d = .38 (N = 36.355; K = 39, Nguyen et al., 2005)

!  Correlation of SJT with cognitive ability drives racial differences found.

!  Moderators !  Reading component !  Presentation format (written vs. video) !  Response instructions

54

Research

Faking, Coaching, & Retest effects

!  Hooper, Cullen, & Sackett (2006) !  Smaller than personality inventories !  d: .08 - .89

!  Moderators

!  Cognitive loading of items !  Transparency of items !  Response instructions !  Type of study design

!  Retest effects (SJTs) < retest effects (cognitive ability tests) (Dunlop et al., 2011; Lievens, Buyse, & Sackett, 2005)

27


Research

Race differences

!  d = .38 (N = 36.355; K = 39, Nguyen et al., 2005)

!  Correlation of SJT with cognitive ability drives racial differences found.

!  Moderators !  Reading component !  Presentation format (written vs. video) !  Response instructions

54

Research

Faking, Coaching, & Retest effects

!  Hooper, Cullen, & Sackett (2006) !  Smaller than personality inventories !  d: .08 - .89

!  Moderators

!  Cognitive loading of items !  Transparency of items !  Response instructions !  Type of study design

!  Retest effects (SJTs) < retest effects (cognitive ability tests) (Dunlop et al., 2011; Lievens, Buyse, & Sackett, 2005)


26—27

28

55

Research

Applicant Perceptions

!  Meta-analysis (Hausknecht et al., 2005)

!  More favorable reactions towards job-related selection procedures

!  Moderators

!  Video SJTs (Chan & Schmitt, 1997)

!  Non-linear SJTs (Kanning et al., 2006)

!  Multimedia SJTs (Richman-Hirsch et al., 2000)

Epilogue

28

55

Research

Applicant Perceptions

!  Meta-analysis (Hausknecht et al., 2005)

!  More favorable reactions towards job-related selection procedures

!  Moderators

!  Video SJTs (Chan & Schmitt, 1997)

!  Non-linear SJTs (Kanning et al., 2006)

!  Multimedia SJTs (Richman-Hirsch et al., 2000)

Epilogue

29

57

Epilogue

Future & Advanced Topics !  Technologically-enhanced SJTs

!  Construct-oriented SJTs !  Trait Activation Theory

!  Cross-cultural issues !  Norming & item banking

!  Alternate form development

[SJTs] seem to represent psychometric alchemy

(adverse impact is down, validity is up), they

seem to assess practically important KSAOs,

and assessees like them

Landy, 2007, p. 418

Epilogue A Famous Quote

28—29

29

57

Epilogue

Future & Advanced Topics !  Technologically-enhanced SJTs

!  Construct-oriented SJTs !  Trait Activation Theory

!  Cross-cultural issues !  Norming & item banking

!  Alternate form development

[SJTs] seem to represent psychometric alchemy

(adverse impact is down, validity is up), they

seem to assess practically important KSAOs,

and assessees like them

Landy, 2007, p. 418

Epilogue A Famous Quote

30

Thank you

More info on http://users.ugent.be/~flievens


!  Becker, T.E. (2005). Development and validation of a situational judgment test of employee integrity. International Journal of Selection and Assessment, 13, 225-232.

!  Bledow, R., & Frese, M. (2009). A situational judgment test of personal initiative and its relationship to performance. Personnel Psychology, 62, 229-258.

!  Bergman, M.E., Drasgow, F., Donovan, M.A., & Henning, J.B. (2006). Scoring situational judgment tests: Once you get the data, your troubles begin. International Journal of Selection and Assessment, 14, 223-235.

!  Bruce, M. M., & Learner, D. B. (1958). A supervisory practices test. Personnel Psychology, 11, 207-216. !  Chan, D., & Schmitt, N. (1997). Video-based versus paper-and-pencil method of assessment is situational judgment tests: Subgroup

differences in test performance and face validity perceptions. Journal of Applied Psychology, 82, 143-159. !  Christian, M. S., Edwards, B. D., & Bradley, J. C. (2010). Situational judgement tests: Constructs assessed and a meta-analysis of

their criterion-related validities. Personnel Psychology, 63, 83-117. !  Cullen, M. J., Sackett, P. R., & Lievens, F. (2006). Threats to the operational use of situational judgment tests in the college

admission process. International Journal of Selection and Assessment, 14, 142-155. !  Chan, D., & Schmitt, N. (2002). Situational judgment and job performance. Human Performance, 15, 233-254. !  Clause, C.C., Mullins, M.E., Nee, M.T., Pulakos, E.D., & Schmitt, N. (1998). Parallel test form development: A procedure for

alternative predictors and an example. Personnel Psychology, 51, 193-208. !  Clevenger, J., Pereira, G.M., Wiechmann, D., Schmitt, N., & Schmidt-Harvey, V.S. (2001). Incremental validity of situational judgment

tests. Journal of Applied Psychology, 86, 410-417. !  Cullen, M.J., Sackett, P.R., & Lievens, F. (2006). Threats to the operational use of situational judgment tests in the college admission

process. International Journal of Selection and Assessment, 14, 142-155. !  Dalessio, A.T. (1994). Predicting insurance agent turnover using a video-based situational judgment test. Journal of Business and

Psychology, 9, 23-32. !  Funke, U., & Schuler, H. (1998). Validity of stimulus and response components in a video test of social competence. International

Journal of Selection and Assessment, 6, 115-123. !  Hooper, A.C., Cullen, M.J., & Sackett, P.R. (2006). Operational threats to the use of SJTs: Faking, coaching, and retesting issues. In

J.A. Weekley & R.E. Ployhart (Eds.), Situational judgment tests: Theory, measurement and application (pp. 205-232). Mahwah, New Jersey: Lawrence Erlbaum Associates.

!  Hunter, D.R. (2003). Measuring general aviation pilot judgment using a situational judgment technique. International Journal of Aviation Psychology, 13, 373-386.

!  Kanning, U.P., Grewe, K., Hollenberg, S., & Hadouch, M. (2006). From the subjects’ point of view – Reactions to different types of situational judgment items. European Journal of Psychological Assessment, 22, 168-176.

!  Konradt, U., Hertel, G., & Joder, K. (2003). Web-based assessment of call center agents: Development and validation of a computerized instrument. International Journal of Selection and Assessment, 11, 184-193.

!  Lievens, F. (2006). International situational judgment tests. In J.A. Weekley & R.E. Ployhart (Eds.) Situational judgment tests: Theory, measurement and application (pp. 279-300). SIOP Frontier Series. Mahwah, NJ: Lawrence Erlbaum Associates.

Useful references


28—29

•Becker, T.e. (2005). Development and validation of a situational judgment test of employee integrity. International Journal of Selection and Assessment, 13, 225-232.

•Bledow, r., & Frese, m. (2009). a situational judgment test of personal initiative and its relationship to performance. Personnel Psychology, 62, 229-258.

•Bergman, m.e., Drasgow, F., Donovan, m.a., & Henning, J.B. (2006). Scoring situational judgment tests: once you get the data, your troubles begin. International Journal of Selection and Assessment, 14, 223-235.

•Bruce, m. m., & Learner, D. B. (1958). a supervisory practices test. Personnel Psychology, 11, 207-216.

•Chan, D., & Schmitt, n. (1997). Video-based versus paper-and-pencil method of assessment is situational judgment tests: Subgroup differences in test performance and face validity perceptions. Journal of Applied Psychology, 82, 143-159.

•Christian, m. S., edwards, B. D., & Bradley, J. C. (2010). Situational judgement tests: Constructs assessed and a meta-analysis of their criterion-related validities. Personnel Psychology, 63, 83-117.

•Cullen, m. J., Sackett, P. r., & Lievens, F. (2006). Threats to the operational use of situational judgment tests in the college admission process. International Journal of Selection and Assessment, 14, 142-155.

•Chan, D., & Schmitt, n. (2002). Situational judgment and job performance. Human Performance, 15, 233-254.

•Clause, C.C., mullins, m.e., nee, m.T., Pulakos, e.D., & Schmitt, n. (1998). Parallel test form development: a procedure for alternative predictors and an example. Personnel Psychology, 51, 193-208.

•Clevenger, J., Pereira, G.m., Wiechmann, D., Schmitt, n., & Schmidt-Harvey, V.S. (2001). incremental validity of situational judgment tests. Journal of Applied Psychology, 86, 410-417.

•Cullen, m.J., Sackett, P.r., & Lievens, F. (2006). Threats to the operational use of situational judgment tests in the college admission

process. International Journal of Selection and Assessment, 14, 142-155.

•Dalessio, a.T. (1994). Predicting insurance agent turnover using a video-based situational judgment test. Journal of Business and Psychology, 9, 23-32.

• Funke, U., & Schuler, H. (1998). Validity of stimulus and response components in a video test of social competence. International Journal of Selection and Assessment, 6, 115-123.

•Hooper, a.C., Cullen, m.J., & Sackett, P.r. (2006). operational threats to the use of SJTs: Faking, coaching, and retesting issues. in J.a. Weekley & r.e. Ployhart (eds.), Situational judgment tests: Theory, measurement and application (pp. 205-232). mahwah, new Jersey: Lawrence erlbaum associates.

•Hunter, D.r. (2003). measuring general aviation pilot judgment using a situational judgment technique. International Journal of Aviation Psychology, 13, 373-386.

•Kanning, U.P., Grewe, K., Hollenberg, S., & Hadouch, m. (2006). From the subjects’ point of view – reactions to different types of situational judgment items. European Journal of Psychological Assessment, 22, 168-176.

•Konradt, U., Hertel, G., & Joder, K. (2003). Web-based assessment of call center agents: Development and validation of a computerized instrument. International Journal of Selection and Assessment, 11, 184-193.

• Lievens, F. (2006). international situational judgment tests. in J.a. Weekley & r.e. Ployhart (eds.) Situational judgment tests: Theory, measurement and application (pp. 279-300). SioP Frontier Series. mahwah, nJ: Lawrence erlbaum associates.

• Lievens, F., Buyse, T., & Sackett, P.r. (2005a). The operational validity of a video-based situational judgment test for medical college admissions: illustrating the importance of matching predictor and criterion construct domains. Journal of Applied Psychology, 90, 442-452.

• Lievens, F., Buyse, T., & Sackett, P.r. (2005b). retest effects in operational selection settings: Development and test of a framework. Personnel Psychology, 58, 981-1007.

USEFUL REFERENCES

30—31

• Lievens, F., & Sackett, P.r. (2006). Video-based versus written situational judgment tests: a comparison in terms of predictive validity. Journal of Applied Psychology, 91, 1181-1188.

• Lievens, F., & Sackett, P.r. (2007). Situational judgment tests in high stakes settings: issues and strategies with generating alternate forms. Journal of Applied Psychology, 92, 1043-1055.

• Lievens, F., Peeters, H., & Schollaert, e. (2008). Situational judgment tests: a review of recent research. Personnel Review, 37, 426-441.

• Lievens, F., Sackett, P.r., & Buyse, T. (2009). The effects of response instructions on situational judgment test performance and validity in a high-stakes context. Journal of Applied Psychology, 94, 1095-1101.

•mcClough, a.C., & rogelberg, S.G. (2003). Selection in teams: an exploration of the Teamwork Knowledge, Skills, and ability test. International Journal of Selection and Assessment, 11, 56-66.

•mcDaniel, m.a., Hartman, n.S., Whetzel, D.L., & Grubb, W.L. (2007). Situational judgment tests, response instructions, and validity: a meta-analysis. Personnel Psychology, 60, 63-91.

•mcDaniel, m.a., morgeson, F.P., Finnegan, e.B., Campion, m.a., & Braverman, e.P. (2001). Predicting job performance using situational judgment tests: a clarification of the literature. Journal of Applied Psychology, 86, 730-740.

•mcDaniel, m.a., & motowidlo, S. (2005). Situational Judgment Tests. Workshop presented at SioP.

•mcDaniel, m.a., & nguyen, n.T. (2001). Situational judgment tests: a review of practice and constructs assessed. International Journal of Selection and Assessment, 9, 103-113.

•mcDaniel, m.a., & Whetzel, D.L. (2005). Situational judgment test research: informing the debate on practical intelligence theory. Intelligence, 33, 515-525.

•mcHenry, J. J. & Schmitt, n. (1994). multimedia testing. in m. G. rumsey & C. B. Walker (eds.), Personnel selection and classification (pp. 193-232). Hillsdale, nJ: erlbaum.

•morgeson, F.P., reider, m.H., & Campion, m.a. (2005). Selecting individuals in team settings: The importance of social skills, personality characteristics, and teamwork knowledge. Personnel Psychology, 58, 583-611.

•motowidlo, S., Dunnette, m.D., & Carter, G.W. (1990). an alternative selection procedure: The low-fidelity simulation. Journal of Applied Psychology, 75, 640-647.

•motowidlo, S.J., Hooper, a.C., & Jackson, H.L. (2006). implicit policies about relations between personality traits and behavioral effectiveness in situational judgment items.Journal of Applied Psychology, 91, 749-761.

•motowidlo, S.J., & Beier, m. e. (2010). Differentiating specific job knowledge from implicit trait policies in procedural knowledge measured by a situational judgment test. Journal of Applied Psychology, 95, 321-333.

•motowidlo, S.J., Crook, a.e., Kell, H.J., & naemi, B. (2009). measuring procedural knowledge more simply with a single-response

• situational judgment test. Journal of Business and Psychology, 24, 281-288.

•nguyen, n.T., Biderman, m.D., & mcDaniel, m.a. (2005). effects of response instructions on faking a situational judgment test.

• international Journal of Selection and assessment, 13, 250-260.

•nguyen, n.T., mcDaniel, m.a., & Whetzel, D.L. (2005, april). Subgroup differences in situational judgment test performance: A metaanalysis. Paper presented at the 20th annual Conference of the Society for industrial and organizational Psychology, Los angeles, Ca.

•o’Connell, m.S., Hartman, n.S., mcDaniel, m.a., Grubb, W.L., & Lawrence, a. (2007). incremental validity of situational judgment tests for task and contextual job performance. International Journal of Selection and Assessment, 15, 19-29.

•olson-Buchanan, J.B., & Drasgow, F. (2006). multimedia situational judgment tests: The medium creates the message. in J.a. Weekley & r.e. Ployhart (eds.), Situational Judgment Tests (pp. 253-278). San Francisco, Jossey Bass.

•olson-Buchanan, J. B., Drasgow, F., moberg, P. J., mead, a. D., Keenan, P. a., & Donovan, m. a. (1998). interactive video assessment of conflict resolution skills. Personnel Psychology, 51, 1-24.

•oswald, F.L., Friede, a.J., Schmitt, n., Kim, B.K., & ramsay, L.J. (2005). extending a practical method for developing alternate test forms using independent sets of items. Organizational Research methods, 8, 149-164.

•


30—31

•oswald, F.L., Schmitt, n., Kim, B.H., ramsay, L.J., & Gillespie, m.a. (2004). Developing a biodata measure and situational judgment inventory as predictors of college student performance. Journal of Applied Psychology, 89, 187-208.

•Peeters, H., & Lievens, F. (2005). Situational judgment tests and their predictiveness of college students’ success: The influence of faking. Educational and Psychological Measurement, 65, 70-89.

•Ployhart, r.e., & ehrhart, m.G. (2003). Be careful what you ask for: effects of response instructions on the construct validity and reliability of situational judgment tests. International Journal of Selection and Assessment, 11, 1-16.

•Ployhart, r.e., Weekley, J.a., Holtz, B.C. & Kemp, C.F. (2003). Web-based and paper-and-pencil testing of applicants in a proctored setting: are personality, biodata, and situational judgment tests comparable? Personnel Psychology, 56, 733-752.

•Smiderle, D., Perry, B. a., & Cronshaw, S. F. (1994). evaluation of video-based assessment in transit operator selection. Journal of Business and Psychology, 9, 3-22.

•Schmitt, n., & Chan, D. (2006). Situational judgment tests: method or construct? in J. Weekley & r.e. Ployhart (eds.), Situational judgment tests (pp. 135-156). mahwah, nJ: Lawrence erlbaum

•Schmitt, n., Keeney, J., oswald, F. L., Pleskac, T., Quinn, a., Sinha, r., & Zorzie, m. (2009). Prediction of 4-year college student performance

using cognitive and noncognitive predictors and the impact of demographic status on admitted students. Journal of Applied Psychology, 94, 1479-1497.

•Stevens, m.J., & Campion, m.a. (1999). Staffing work teams: Development and validation of a selection test for teamwork. Journal of Management, 25, 207-228.

•Such, m.J., & Schmidt, D.B. (2004, april). Examining the effectiveness of empirical keying: A cross-cultural perspective. Paper presented at the 19th annual Conference of the Society for industrial and organizational Psychology, Chicago, iL.

•Wagner, r.K., & Sternberg, r.J. (1985). Practical intelligence in real world pursuits: The role of tacit knowledge. Journal of Personality and Social Psychology, 49, 436-458.

•Weekley, J.a., & Jones, C. (1997). Video-based situational testing. Personnel Psychology, 50, 25-49.

•Weekley, J.a., & Jones, C. (1999). Further studies of situational tests. Personnel Psychology, 52, 679-700.

•Weekley, J.a., & Ployhart, r.e. (2006). Situational judgment tests: Theory, measurement and application. San Francisco, Jossey Bass.

•Whetzel, D.L., & mcDaniel, m.a. (2009). Situational judgment tests: an overview of current research. Human Resource Management Review, 19, 188-202.

32—3332—33

Evaluations of situational judgments tests for high stakes selection.— Prof. Fiona Patterson

Departamento de Psicología de la Universidad de Cambridge, reino Unido y directora del Work Psychology Group.

32—3332—33

Evaluations of situational judgement tests

for high stakes selection:

Research & practice in the healthcare professions

Professor Fiona Patterson

Work Psychology Group & University of Cambridge

Madrid, June 2013

Overview • Selecting doctors in the UK National Health Service • Selection methods – Low fidelity: Knowledge test, situational judgement test (SJT) – High fidelity: Assessment Centre (AC)

• Predictive validity studies • Low fidelity simulations are cost-effective to develop,

administer & score, but how do they compare to high fidelity ACs?

• Current research & implications for policy & practice

UK Medical Training & Career Pathway

Selection Gateway 2

Selection Gateway 1

Core Specialty Training

Under-graduate medical school training

Specialty Training

FT1 FT2 CCT &

eligible for Consultant

Grade

Higher Specialty Training

Foundation Training (FT)

2 Years

Specialty Training (ST) 8 Years

Selection Gateway 3

GP Specialty Training

3 years

34—35

“Work for me, son – I knew your father.”

1970

“ Fill out the application form for HR

and the job is yours, mate.”

1980 1990

“It isn’t an interview – just

an informal chat, sweetie.

Just a formality.”

Selection in medicine through the ages…

Help!

• 26,000 applicants for 8,000 medical school places

• 8,000 medical students apply for their first post

• 10,000 speciality applicants

• 27,000 + interviews

• Weeks of offering, rejecting, cascading

1000s Consultant hours

What non-cognitive attributes are important to be an effective clinician? What methods are available to test these in high stakes selection? Should we use personality testing?

Given the costs, beyond some basic assessment,

is a lottery the best option?

Key research questions

evaluations of situational judgments tests for high stakes selection.

34—35

Why not use a lottery system?

GP Specialty selection (selection gateway 3) N= 6,500 applicants per year for 3,000 training posts


Selection Gateway 2

Selection Gateway 1



Specialty Training

FT1 FT2 CCT &


Grade



2 Years


Selection Gateway 3


3 years

36—37

Job analysis of the GP role • GP selection system is based on

multi-source, multi-method job analysis Patterson et al, BJGP, 2000; Patterson et al, BJGP, 2013

• Empathy & sensitivity • Communication skills • Problem-solving • Professional integrity • Coping with pressure • Clinical expertise

British Journal of General Practice, May 2013

Previously…

Shortlisting Interview

ElectronicApplicationProcess

National Regional

GP Selection

Assessment Centre UsingSimulated Consultations

Longlisting Selection Tests Assessment Centre (AC)

Clinical Problem solving testCPS

Situational Judgement testSJT

Ranking

Matching to Region

Foundation Competency

National panel


36—37

Clinical Problem Solving (CPS) Knowledge Test • Applying clinical knowledge in relevant contexts, e.g. diagnosis, management

• Item type single best answer/ multiple choice • Operational test: 100 items, duration 90 minutes

Reduced Vision A. Basilar migraine F. Central retinal vein occlusion B. Cerebral tumour G. Optic neuritis (demyelinating) C. Cranial arteritis H. Retinal detachment D. Macular degeneration I. Tobacco optic neuropathy E. Central retinal artery occlusion

For each patient below select the SINGLE most likely diagnosis from the list above. Each option may be selected once, more than once or not at all.

1. A 75 year old man, who is a heavy smoker, with a blood pressure of 170/105, complains of floaters in the left eye for many months and flashing lights in bright sunlight. He has now noticed a "curtain" across his vision.

Situational Judgement Test (SJT) • Professional dilemmas focusing on non-cognitive attributes in

complex interpersonal scenarios (empathy, integrity, coping with pressure, teamworking)

• Item writing by subject matter experts, concordance panel to agree scoring key

• Cognitively-oriented (‘What should you do?’) • Item types: rank 4/5 options, choose best 2/3 • Operational tests comprise 50 items, in 90 minutes

Example SJT item

You are reviewing a routine drug chart for a patient with rheumatoid arthritis during an overnight shift. You notice that your consultant has inappropriately prescribed methotrexate 7.5mg daily instead of weekly.

Rank in order the following actions in response to this situation (1= Most appropriate; 5= Least appropriate)

A Ask the nurses if the consultant has made any other drug errors recently

B Correct the prescription to 7.5mg weekly C Leave the prescription unchanged until the consultant ward round

the following morning D Phone the consultant at home to ask about changing the prescription E Inform the patient of the error

38—39

Predictive Validity Results

GP Selection: Validation studies

Study 1. Supervisor ratings after 1 year into training

Study 2. End-of-training outcomes (licensure exam) after 3 years

Study 3. Structural (theoretical) model to evaluate the incremental validity for each selection method

Study 1: Correlations between the selection methods & job performance after 1 year

N=196 Mean SD 1. 2. 3.

Selection methods (Predictors)

1. Clinical problem-solving test 78.88 9.02

2. Situational judgement test 637.87 34.31 .50

3. Assessment centre 3.32 0.39 .30 .43

Outcome variable

4. Supervisor ratings 4.63 0.73 .36 (.54)

.37 (.56)

.30 (.50)

Note. Correlations between parentheses were corrected for multivariate range restriction. Correlations are significant at p < .01


38—39

Study 2: Correlations between the selection methods & end-of-training licensure assessments

N=2292 Mean SD 1. 2. 3. 4.

Selection methods (Predictors)

1. Clinical problem-solving test 80.08 8.14 2. Situational judgement test 640.13 31.66 .40 3. Assessment centre 3.34 0.36 .24 .32 Outcome variables 4. End of training applied knowledge test

0.26 0.90 .73 (.85)

.43 (.69)

.24 (.41)

5. End of training clinical skills exam (Simulated surgery)

0.20 0.80 .38 (.55)

.43 (.57)

.32 (.41)

.41

Note. Correlations between parentheses were corrected for multivariate range restriction. Correlations are significant at p < .01

Study 3. Structural/theoretical model showing selection methods & their link to job performance Lievens & Patterson (2011)

Significant incremental validity offered by the AC

SJT fully mediates the effects of declarative knowledge on job performance. The AC partially mediates the effects of the SJT

Theoretical Model Underlying SJTs in GP Selection

General Experience

Implicit Trait Policies

Specific Job Experience

Specific Job Knowledge

Job/Training Performance

Procedural Knowledge Captured

by the SJT

40—41

Current Research into Medical & Dental School Admissions

UK Clinical Aptitude Test (UKCAT)

http://www.ukcat.ac.uk/

5 subtests

• Verbal, numerical, abstract reasoning & decision analysis

• Non-cognitive analysis using an SJT - targeting empathy, integrity & team involvement


Selection Gateway 2

Selection Gateway 1



Specialty Training

FT1 FT2 CCT &


Grade



2 Years


Selection Gateway 3


3 years

SJT Test Specification Content • Scenarios based in either a clinical setting or during

education/training for a medical/dental career • Third party perspective to increase breadth of available scenarios Response Format (rating using a 4 point scale) • Rate the appropriateness of a series of options from ‘very

appropriate’ to ‘very inappropriate. • Rate the importance of a series of options from ‘very important’ to

‘not important at all’


40—41

Example UKCAT SJT items A consultation is taking place between a senior doctor and a patient; a medical student is observing. The senior doctor tells the patient that he requires some blood tests to rule out a terminal disease. The senior doctor is called away urgently, leaving the medical student alone with the patient. The patient tells the student that he is worried he is going to die and asks the student what the blood tests will show. How appropriate are each of the following responses by the medical student in this situation? Q1 Explain to the patient that he is unable to comment on what the tests will

show as he is a medical student Q2 Acknowledge the patient’s concerns and ask whether he would like them to

be raised with the senior doctor Q3 Suggest to the patient that he poses these questions to the senior doctor

when he returns Q4 Tell the patient that he should not worry and that it is unlikely that he will die

UKCAT SJT Evaluation • Reliability of a 70 item test with similar quality items estimated

(α=.75 to .85) • Candidate reactions show high face validity (significantly more

than the cognitive subsections of UKCAT)

• Content of SJT relevant for med/dental applicants = 70% • Content of the SJT is fair to med/dental applicants = 63%

Group differences & content validity • Gender: Females outperform males (0.2 SD) • Ethnicity: White candidates performed better (0.3SD) • Occupation & Employment Status: those in the higher occupational

classes (i.e. Managerial/Professional Occupations) do not always score higher than those in lower classes - in some cases those from lowest occupational groups, received the highest mean score.

• SJT correlates with other subtests (approx r=0.28) indicating some shared variance between the tests. Since a large amount of variance is not explained, the SJT is assessing different constructs to the other tests.

42—4342—43

Implications for policy & practice • Strong predictive validity translates into significant gains in utility • Focus not on ‘how much validity’ does a selection method add but

more ‘valid for what’ • Closer attention to the criterion constructs targeted as high-fidelity

simulations show incremental validity over low-fidelity simulations for predicting interpersonal job performance dimensions

• Bespoke job analysis is the cornerstone to effective selection • Positive candidate reactions (Patterson et al, 2010)

• Political validity is an important concept in this setting (Patterson et al, 2012)

Thank you

[email protected] Some useful links http://www.workpsychologygroup.com/ http://www.gprecruitment.org.uk/ http://www.isfp.org.uk/ http://www.ukcat.ac.uk/about-the-test/behavioural-test/ http://www.agpt.com.au/ http://www.gmc-uk.org/about/research/14400.asp


42—4342—43

44—4544—45

Situational Judgment Test in EPSO open competitions for the European Institutions from 2010 to present.— Mr. Stefan Meyer

ePSo (european Personnel Selection office).

44—4544—45

1

Stefan Meyer, European Personnel Selection Office

Madrid, 26 June 2013

Situational Judgment Tests in EPSO open competitions for the European Institutions

The European Union

Commission

Council

Parliament

Court of Justice

Court of Auditors

Economic & Social Committee

Committee of the Regions

Ombudsman

Data Protection Supervisor

The EU Institutions

Posts in 2012 European Parliament 6655 Council of the European Union 3153 European Commission 25478 Court of Justice 1952 European Court of Auditors 887 European Economic and Social Committee 724 Committee of the Regions 531 European Ombudsman 66 European Data Protection Supervisor 43 European External Action Service 1670 Total 41159

European External

Action Service4% Others

3%

European Parliament

16%

Council of the

European Union

8%

European Commission

62%

European Court

of Auditors2%

Court of Justice

5%

46—47

Entry Streams

•  Administrators (AD) e.g. policy officers, lawyers, auditors, scientific officers, translators, interpreters, communication & press officers …

Qualification: University degree (bachelor)

•  Assistants (AST) e.g. secretaries, HR assistants, financial assistants, conference technicians …

Qualification: Secondary education (at least) and relevant professional experience

5

Where in competition procedure?

CBT Admission Assessment centre

Reserve list Registration

6

EPSO Development Programme (2008 – 2011)

Redesign of the CBT pre-selection phase: •  Keep verbal reasoning •  Keep numerical reasoning •  Skip EU knowledge •  Introduce abstract reasoning tests •  Introduce Situational Judgement Tests •  Introduce professional skills testing,

including linguistic skills •  New linguistic regime as of 2011: cognitive

tests in 23 languages

Situational Judgment Test in ePSo open competitions for the european institutions from 2010 to present.

46—47

7

New EPSO admission phase

CBT

Professional / Linguistic Skills (where appropriate)

Verbal & Numerical reasoning

- In 23 languages as of 2011!

Situational Judgement Tests

(in EN, FR, DE; where appropriate)

Abstract Reasoning

8

Situational Judgment Tests

9

EPSO Development Programme

•  EDP, action No. 8: The introduction of situational/ behavioural testing based on a well-founded competency framework

Why? •  Good indicators for job performance •  Widely used and perceived as relevant

+ fair

48—49

10

SJT competencies* (from the EU competency framework)

Analysing &

Problem solving

Delivering Quality &

Results

Prioritising& Organising

Resilience

Working with Others

!  Identifies the critical facts in complex issues and develops creative and practical solutions

!  Takes personal responsibility and initiative for delivering work to a high standard of quality within set procedures

!  Prioritises the most important tasks, works flexibly and organises own workload efficiently

!  Remains effective under a heavy workload, handles organisational frustrations positively and adapts to a changing work environment !  Works co-operatively with others in teams and across organisational boundaries and respects differences between people

11

SJT: features

–  SJT evaluate workplace-related behaviour –  Target groups: AD5 and AST3 annual cycle of

competitions –  Administered @ CBT stage –  Score report used as expert input @ AC stage –  20 items test in 30 mins. in either EN, FR or DE

=> NOT a speed test! –  Tests developed in cooperation with expert external

contractor –  First launch: AD cycle 2010 (non-eliminatory)

12

Multi-stage TD process for SJT

1.  Creation of draft item-models in EN 2.  Review by experienced SME (contractor) =>

amendments/modifications (if applicable) 3.  Statistical trials => selection of item models 4.  Review by EPSO quality board in view of

–  Organisational fit –  Degrees of ambiguety –  Translation/localisation (and other)

5.  Translation into FR and DE 6.  CBT upload => ROLL OUT


48—49

13

Which are the most and least effective options? You work as part of a technical support team that produces work internally for an organisation. You have noticed that often work is not performed correctly or a step has been omitted from a procedure. You are aware that some individuals are more at fault than others as they do not make the effort to produce high quality results and they work in a disorganised way.

a.  Explain to your team why these procedures are important and what the consequences are of not performing these correctly.

b.  Try to arrange for your team to observe another team in the organisation who produce high quality work.

c.  Check your own work and that of everyone else in the team to make sure any errors are found.

d.  Suggest that the team tries many different ways to approach their work to see if they can find a method where fewer mistakes are made.

M

L

14

Cdts sentiment towards SJT

15

SJT cumulative score distributions

50—51

16

To sum up •  No evidence of bias by language version

•  No evidence of bias by gender or age •  Good psychometric properties for the tests

–  Reliabilities found to exceed 0.7 for the short forms –  This has been confirmed for the longer 20 items tests

•  Score ranges showed that the tests would discriminate between candidates in an effective but fair way

…. and candidates actually like SJT!

17

Thank you for your attention!


50—51

52—5352—53

Medición de competencias mediante Modelos de Diagnóstico Cognitivo: aplicación a un Test de Juicio Situacional.— Prof. Julio Olea

Catedrático de Psicología de la Universidad autónoma de madrid y codirector de la cátedra Uam-iiC modelos y aplicaciones psicométricos (maP).

52—5352—53

Medición de competencias mediante Modelos de Diagnóstico Cognitivo:

aplicación a un Test de Juicio Situacional

Pablo E. García (IIC)

Julio Olea (UAM)

Jimmy de la Torre (Rutgers U.)

Índice de la presentación

– Ejemplo: estimación de la competencia

matemática.

– MDC: modelos, proceso de aplicación y aportaciones previsibles para la medición de competencias.

– Estudio empírico: objetivos, procedimiento y resultados al aplicar un MDC a un TJS.

Ejemplo: Competencia matemática de un niño de 6 años

• ¿Tiene adquiridas las competencias de suma y

de resta de números naturales?

• 4 posibles estados latentes: a) No suma, no resta (0, 0) b) Suma, no resta (1, 0) c) No suma, resta (0, 1) d) Suma, resta (1, 1)

54—55medición de competencias mediante modelos de Diagnóstico Cognitivo: aplicación a un Test de Juicio Situacional.

Test Respuestas de un niño

Ítem 1: 3+4= 7

Ítem 2: 7-2= 9

Ítem 3: 7+6-3= 16

Test Respuestas de un niño

Ítem 1: 3+4= 7

Ítem 2: 7-2= 9

Ítem 3: 7+6-3= 16

¿Qué estado latente es más probable? a) No suma, no resta (0, 0) b) Suma, no resta (1, 0) c) No suma, resta (0, 1) d) Suma, resta (1, 1)

El MDC más simple: modelo DINA

• Matriz Q:

• Asume dos parámetros para cada ítem: • s: slipping (desliz). P. ej. 0.1 • g: guessing (acierto por azar). P. ej. 0.2

• Asume que es preciso tener todas las competencias requeridas para

acertar un ítem.

54—55

Probabilidades de acierto

Probabilidades de acierto

ESTADO LATENTE ( ) Ítem 1

(1,0)

Ítem 2

(0,1)

Ítem 3

(1,1)

No suma, no resta (0, 0) 0.2 0.2 0.2

Suma, no resta (1, 0) 0.9 0.2 0.2

No suma, resta (0, 1) 0.2 0.9 0.2

Suma, resta (1, 1) 0.9 0.9 0.9

Verosimilitud

Probabilidades para el niño

ESTADO

LATENTE ( )

Ítem 1

Ítem 2

Ítem 3

Verosimi-

litud

No suma, no resta (0, 0) 0.2 0.8 0.8 0.032

Suma, no resta (1, 0) 0.9 0.8 0.8 0.144

No suma, resta (0, 1) 0.2 0.1 0.8 0.016

Suma, resta (1, 1) 0.9 0.1 0.1 0.009

Verosimilitud

Probabilidades para el niño

ESTADO

LATENTE ( )

Ítem 1

Ítem 2

Ítem 3

Verosimi-

litud

No suma, no resta (0, 0) 0.2 0.8 0.8 0.032

Suma, no resta (1, 0) 0.9 0.8 0.8 0.144

No suma, resta (0, 1) 0.2 0.1 0.8 0.016

Suma, resta (1, 1) 0.9 0.1 0.1 0.009

56—57

Diferentes MDC VARIEDAD DE MODELOS (Rupp, Templin & Henson, 2010):

• No compensatorios:

– DINA: Deterministic-input, noisy-and-gate (Junker & Sijtsma, 2001). • Compensatorios:

– DINO: Deterministic-input, noisy-or-gate (Templin & Henson, 2006): s es la probabilidad de fallo cuando “al menos” un atributo está presente.

• Modelos generales: – G-DINA (de la Torre, 2011).

PECULIARIDADES: • Modelos multidimensionales. • Naturaleza confirmatoria: podemos obtener indicadores de ajuste para aplicar un

modelo u otro. • Novedosos, en relación a otros modelos psicométricos (TCT, TRI): mayor

información “diagnóstica”.

Proceso de aplicación de un MDC

Jueces

expertos

[0 1 0 1 1] K2

TJS para la medición de competencias

• Problemas habituales de validez (Christian, Edwards & Bradley, 2010; Ployhart & MacKenzie, 2011). – Evidencias sobre validez de contenido: ¿Qué miden

exactamente los TJS? ¿Una o diferentes competencias? Normalmente se extrae una única puntuación.

– Evidencias sobre la estructura interna: ¿Cómo comprobar que las competencias están realmente reflejadas en los ítems o situaciones?

– Evidencias sobre relaciones con otras variables. Estudios de validez referida al criterio inconsistentes. Estudios prometedores de validez predictiva (Lievens & Sackett, 2012).

medición de competencias mediante modelos de Diagnóstico Cognitivo: aplicación a un Test de Juicio Situacional.

56—57

TJS para la medición de competencias (Lievens & Sackett, 2012)

PREDICTORES

(examen de admisión)

CRITERIOS ( )

Calificaciones en las prácticas (7años)

Rendimiento en el trabajo (9 años)

Test de capacidades cognitivas y de conocimientos científicos

.13(*) .12

Test de contenidos médicos -.02 -.17

TJS en video (interpersonal skills) .22(*) .23(*)

¿Qué pueden aportar los MDC a la medición de competencias?

Evidencias sobre el contenido

Evidencias sobre la estructura interna y los procesos de

respuesta

Evidencias sobre relaciones con otras variables

Ajuste del modelo a los datos

Matriz Q

??? Habrá que comprobarlo

Estudio empírico: objetivos

• Aplicar los MDC a la medición de competencias profesionales a través de TJS.

• Pregunta principal: ¿qué competencias podemos juzgar que tienen o no tienen los evaluados?

• Preguntas adicionales: – ¿Qué competencias están incorporadas en las situaciones? – ¿Son todas relevantes? – ¿Se requieren todas las competencias incluidas en una situación

para emitir la respuesta apropiada? – ¿Se relacionan las medidas categóricas que proporciona el MDC

con otras variables?

58—59

Participantes e instrumentos

ÍTEM 4: Durante una reunión con su equipo de trabajo y algunos responsables de otras áreas, tras exponer las cualidades del proyecto en desarrollo, uno de los responsables contesta: “Encuentro que su proyecto presenta algunas carencias”. ¿Qué respondería? a) No estoy de acuerdo, pero explíqueme exactamente qué tipo de carencias y

examinemos todos los puntos sobre los que tenga cualquier duda. b) Probablemente es que no hemos profundizado lo suficiente en las ventajas.

Repasemos de nuevo cada uno de los puntos del proyecto. c) Puede ser, es un tema sobre el que estamos trabajando y al que esperamos darle

pronto solución. Aún así, será rentable de igual modo.

Participantes: 485 empleados de una entidad bancaria española (programa de desarrollo interno). Se aplicó el TJS y un test de personalidad (Big Five).

TJS informatizado de 30 ítems (perteneciente a la batería eValue del IIC).Cada ítem comienza con un incidente crítico, seguido de una pregunta y tres opciones.

Definición de la matriz Q

Competencias Factores Big Five

Decidir e iniciar acciones

Relacionarse y establecer redesInteractuar y

presentarExtraversiónPersuadir e influenciar

Presentar y comunicar información

Seguir instrucciones y procedimientos

Adaptarse y responder al cambio Adaptarse y

aguantar

Estabilidad

emocionalAguantar la presión y contratiempos

Librería de competencias Great-Eight (SHL Group; Kurz & Bartram, 2002; Bartram, 2005). Tres niveles:

1. 8 factores generales. 2. 20 competencias 3. 112 componentes.

7

Definición de la matriz Q • 4 expertos en Psicología organizacional especificaron la

Matriz-Q: – Se les proporcionó varios indicadores para cada competencia. – Valoraron, para cada ítem: ¿qué competencias son necesarias

para dar la respuesta considerada como más adecuada? – La tarea se llevó a cabo en 3 fases:

• Fase 1: tarea individual. • Fase 2: posible reconsideración cuando conocen los resultados de la

fase 1. • Fase 3: grupo de discusión para intentar llegar a mayores acuerdos.

– Una competencia (Relacionarse y establecer redes) fue asignada sólo a 4

ítems.


58—59

Definición de la matriz Q • ¿Cuál de las matrices Q?

– Se valoró estadísticamente el ajuste de diferentes posibles matrices. – Chen, de la Torre & Zhang (en prensa): el índice BIC (Bayesian

Information Criterion; Schwarz, 1976) permite elegir la mejor matriz, asumiendo como cierto el modelo G-DINA.

= + ( )

Matriz-Q (7) BIC Matriz-Q (6) BIC

Unanimidad 16.115,88 Unanimidad 15.782,75 Al menos dos 16.169,48 Al menos dos 15.749,79 Al menos uno 16.300,34 Al menos uno 15.840,37

4 ítems eliminados

1 competencia eliminada

Definición de la matriz Q

1. Persuadir e influenciar 2.Presentar y comunicar información 3.Seguir instrucciones y procedimientos

4.Adaptarse y responder al cambio 5.Aguantar la presión y contratiempos 6.Decidir e iniciar acciones

Item Competency Item Competency 1 2 3 4 5 6 1 2 3 4 5 6 1 0 0 1 1 1 0 14 0 0 1 1 0 0 2 0 1 0 1 1 0 15 1 1 0 0 0 1 3 1 1 0 0 0 1 16 0 1 0 0 0 0 4 1 0 0 0 1 0 17 0 0 1 0 0 0 5 0 1 0 0 0 0 18 1 0 0 0 0 0 6 0 0 1 1 1 0 19 1 1 0 0 0 0 7 1 0 0 0 0 0 20 1 1 0 0 0 1 8 0 0 1 1 1 0 21 0 0 1 1 0 0 9 0 0 1 1 1 0 22 0 0 1 1 1 0 10 1 1 0 0 0 0 23 0 0 1 1 1 0 11 0 0 0 1 0 1 24 1 1 0 0 0 1 12 1 1 0 0 0 1 25 1 1 0 0 0 1 13 0 0 1 0 1 1 26 1 1 0 0 0 1

ÍTEM 4: Durante una reunión con su equipo de trabajo y algunos responsables de otras áreas, tras exponer las cualidades del proyecto en desarrollo, uno de los responsables contesta: “Encuentro que su proyecto presenta algunas carencias”. ¿Qué respondería? a) No estoy de acuerdo, pero explíqueme exactamente qué tipo de carencias y examinemos todos los puntos sobre los que tenga

cualquier duda.

Selección del MDC • Ajuste relativo:

– Una vez seleccionada la Matriz-Q, se comparó el modelo G-DINA con modelos más restrictivos (y parsimoniosos): DINA (no-compensatorio) y DINO (compensatorio).

– En la medida en que son modelos anidados, se empleó la razón de verosimilitudes (Likelihood Ratio) para valorar si la pérdida de ajuste al emplear un modelo más sencillo era estadísticamente significativa. Así fue.

• Se seleccionó en consecuencia el modelo G-DINA (de la Torre, 2011).

60—61

Estimación de perfiles de competencias

• Nº de perfiles posibles: • Los más frecuentes:

6422 6K

% Competency

1 2 3 4 5 6

9.46 1 1 0 0 1 1

8.25 1 0 0 0 1 0

8.08 0 1 0 1 0 0

7.93 1 1 1 0 1 1

7.05 1 1 1 1 1 1

1. Persuadir e influenciar 2.Presentar y comunicar información 3.Seguir instrucciones y procedimientos 4.Adaptarse y responder al cambio 5.Aguantar la presión y contratiempos 6.Decidir e iniciar acciones

Evidencias de validez: relaciones con otras variables

Adaptarse y aguantar Estabilidad emocional (promedio)

Posee la competencia 69,26

No posee la competencia 64,59

T=-3,37; punilateral=0,0005

Interactuar y presentar Extraversión (promedio)



T=-2,52; punilateral=0,006

Adaptarse y aguantar Estabilidad emocional (promedio)



Interactuar y presentar Extraversión (promedio)




60—61

MUCHAS GRACIAS

THANK YOU VERY MUCH

CONCLUSIONES: ¿Qué pueden aportar los MDC a la medición de competencias?

Evidencias sobre el contenido

Evidencias sobre la estructura interna y los procesos de

respuesta

Evidencias sobre relaciones con otras variables

- Permiten especificar los contenidos incluidos en los ítems.

- Podemos quedarnos con las competencias realmente válidas. -El modelo que mejor ajusta nos proporciona información sobre los procesos de respuesta.

- Convendría estudiar si proporcionan mejores pronósticos del rendimiento en el trabajo.

62—6362—63

RESUMEN POSTERS

62—6362—63

Competencias de empleabilidad: diseño de un test situacional.

— Laura Arnau Sabate1, Josefina Sala Roca1, Mercè Jariot2, Teresa Eulalia Marzo3, Adrià Pagès2

reSUmen

Las competencias de empleabilidad son competencias básicas necesarias para obtener y mantener un trabajo independientemente del tipo y sector de ocupación. Son transversales y transferibles a otras situaciones o contextos personales y profesionales.

nuestro grupo ha desarrollado una propuesta de competencias de empleabililidad (autoor-ganización, contrucción del proyecto formativo-profesional, toma de decisiones y resolu-ción de problemas, trabajo en equipo, comunicación, perseverancia, flexibilidad y respon-sabilidad) validado por un grupo de expertos en una primera fase (arnau et al, 2013) y que posteriormente ha sido validado con profesionales de los diferentes sectores económicos de la población (trabajo pendiente de publicación).

Se ha diseñado un test situacional a partir de la creación de 5 historias ambientadas en diferentes contextos de la vida cotidiana (de amigos, escolar, familiar, más formales o in-formales). en estas historias se plantean situaciones a las que el sujeto debe responder como va a actuar frente a esa situación. estas situaciones requieren de un buen nivel de competencia para ser afrontadas adecuadamente. Se contó con un panel de expertos para la elaboración de las historias y situaciones.

Para elaborar las respuestas, se realizaron entrevistas a 115 jóvenes de edades entre 12 y 18 años, que fueron transcritas y sometidas a análisis de contenido. el panel de expertos seleccionó de entre las respuestas dadas 5 respuestas que podrían ser indicadoras de diferentes niveles de competencia y que asimismo asegurarían suficiente variabilidad.

Posteriormente otro panel de 11 expertos distintos está validando la asignación de cada una de las situaciones como indicador de las competencias, y en una segunda fase se pedirá a estos expertos que asignen un valor a cada respuesta en función del nivel de competencia.

Una vez construido el test será validado para comprobar sus propiedades psicométricas y revisar su estructura interna.

1 Departament de Pedagogia Sistemàtica i Social. Universitat autònoma de Barcelona2 Departament de Pedagogia aplicada. Universitat autònoma de Barcelona3 Pere Tarrés. Universitat ramón Llull

64—65

1 Presentador: Juan Carlos Pérez morán

Correo electrónico: [email protected]

instituto de investigación y Desarrollo educativo Universidad autónoma de Baja California

T. celular. 044 (664) 3851253 · T. (646) 175-07-33 ext. 64508 · F. (646) 174-20-60

Análisis de la estructura cognitiva del área de habilidades cuantitativas del exhcoba mediante los modelos LLTM y LSDM.

— Juan Carlos Pérez Morán1, Norma Larrazolo Reyna y Eduardo Backhoff Escudero Instituto de Investigación y Desarrollo Educativo Universidad Autónoma de Baja California

reSUmen

en el presente estudio se integran aspectos de la psicología cognitiva y la psicometría (rupp y mislevy, 2006) para el análisis de la estructura cognitiva del área de Habilidades Cuanti-tativas (HC) del examen de Habilidades y Conocimientos Básicos (eXHCoBa) (Backhoff, Tirado, y Larrazolo, 2001), mediante el Linear Logistic Latent Trait model (en inglés, LLTm) de Fischer (1973, 1997) y el Least Squares Distance method (en inglés, LDSm) propuesto por Dimitrov (2007). Para alcanzar dicho propósito, un grupo de expertos propuso catorce operaciones para resolver los ítems del área HC apoyándose en los resultados de reportes verbales concurrentes y retrospectivos realizados a un grupo de estudiantes de educación secundaria. además, se analizó la estructura cognitiva determinada por los expertos y se evaluó la dimensionalidad y el ajuste de los ítems a los modelos rasch y LLTm. Los resulta-dos mostraron que los datos ajustan moderadamente al modelo LLTm con una matriz Q de catorce atributos y que un refinamiento de las operaciones para resolver los ítems podría explicar mejor sus fuentes de dificultad y el rendimiento de los examinados en la prueba. además, la aplicación del LSDm mostró que es necesario mejorar la validación de los atri-butos cognitivos requeridos en la solución de ítems del área de HC del eXHCoBa.

Palabras clave: modelo LLTm, modelo rasch, matriz Q, eXHCoBa.

64—65

reSUmen

La competencia iniciativa-innovación es una competencia transversal definida en la mayoría de los modelos de competencias de mayor uso tanto en nuestro país como en el contexto internacional (Bartram, 2002; alles, 2002, Lombardo, 2005).

en este trabajo se presenta el diseño, construcción y análisis de un test situacional orienta-do a la evaluación de esta competencia. en la prueba diseñada el evaluado debe imaginar que forma parte de un equipo de trabajo e interactuar con el resto de componentes a través del correo electrónico. Los diferentes mensajes de correo electrónico que recibe plantean situaciones donde es posible manifestar los diferentes comportamientos asociados a la competencia objeto de medida. además, el evaluado tiene la posibilidad de enviar mensajes por iniciativa propia.

aplicada a una muestra de 261 profesionales de una entidad financiera de ámbito multina-cional los resultados muestran una buena capacidad de discriminación de la prueba y una baja consistencia interna (alpha=.52) -en línea con otros test situacionales-.

Midiendo la competencia Iniciativa-Innovación mediante Test: diseño, construcción y análisis psicométrico inicial de un Test Informatizado de Juicio Situacional.

— Sonia Rodríguez, Beatriz Lucía, David Aguado Instituto de Ingeniería del Conocimiento

66—67

Cheating en administración de Test por Internet. Análisis del Test de Verificación eCat.

— David Aguado1, Julio Olea2, Vicente Ponsoda2 y Francisco Abad2

reSUmen

el uso de internet ha modificado en los últimos tiempos las prácticas de reclutamiento y se-lección desarrolladas por las compañías y es común que tanto el proceso de reclutamiento como los procesos de evaluación inicial se realicen a través de internet en lo que se conoce como Unproctored Internet Testing (UiT). a pesar de las grandes ventajas asociadas al UiT en términos de costes y flexibilidad es una práctica cuanto menos controvertida debido a problemas relacionados con el engaño (cheating). Para controlarlo, la recomendación de la International Test Comission es la de realizar una evaluación en un entorno controlado y comparar las puntuaciones en la condición UiT y la controlada para identificar a las perso-nas que engañan. el objetivo del trabajo que se presenta es analizar los resultados que se obtienen al implantar un procedimiento específico de verificación sobre un Test adaptativo informatizado para la evaluación del nivel de inglés: eCat; aplicado en un proceso real de selección de personal. Se evaluó a una muestra de 417 candidatos a puestos de trabajo como ingenieros en una multinacional española del sector energético. Los resultados iniciales muestran como el procedimiento implementado es eficaz en la detección del cheating detectándose que 44 personas (10.55%) habían utilizado algún tipo de ayuda adicional en el entorno UiT.

1instituto de ingeniería del Conocimiento2Departamento de Psicología Social y metodología. Universidad autónoma de madrid.

66—67

reSUmen

el BaT-7 es una nueva batería que permite estimar la inteligencia y evaluar 8 aptitudes cog-nitivas. Se estructura en 3 niveles de dificultad creciente y está enfocada a la evaluación de escolares, universitarios y adultos.

Método: La muestra de tipificación estuvo compuesta por 4.263 escolares y 1.507 adultos con diferente grado de formación. Se ha utilizado un modelo logístico unidimensional de 3 parámetros y se ha estudiado el ajuste mediante los residuos estandarizados y el estadístico chi-cuadrado. La fiabilidad se ha analizado utilizando el coeficiente alfa ordinal y las funcio-nes de información. Para el estudio de la validez se desarrolló un modelo aFC multigrupo.

Resultados: el modelo Tri de 3 parámetros mostró un adecuado ajuste en todos los ítems. Los valores de fiabilidad oscilaron entre 0,79 y 0,91 para los tests y entre 0,91 y 0,97 para los índices de inteligencia. el modelo estructural basado en la teoría CHC presentó un buen ajuste a los datos empíricos.

Discusión: el BaT-7 supone una medida informativa y altamente fiable para la evaluación de las aptitudes cognitivas. igualmente, mediante su aplicación puede obtenerse una esti-mación bastante razonable de la capacidad general (g), de la inteligencia fluida (Gf) y de la inteligencia cristalizada (Gc).

Batería de aptitudes de TEA: Fiabilidad y evidencias de validez.

— David Arribas Águila Dpto. de I+D+i de TEA Ediciones

Correspondencia: [email protected]/ Fray Bernardino Sahagún, 24. 28036 madrid

68—69

Papel del psicólogo del trabajo en el reclutamiento y selección de personas.

— Gloria Castaño, Vicente Ponsoda, M. Ángeles Romeo, Pedro Sanz, Ana Sanz, José Manuel Chamorro, C. Manuel Leal, Yolanda Peinador, Jesús Martínez, Laura Alonso, M. Ely Iglesias, Francisco Álvarez y Luis Picazo Grupo de Trabajo de Buenas Prácticas en Reclutamiento y Selección del Colegio Oficial de Psicólogos de Madrid

reSUmen

en este estudio se presentan los resultados de una encuesta realizada por el Grupo de Tra-bajo de Buenas Prácticas en reclutamiento y Selección del Colegio oficial de Psicólogos de madrid. Su principal objetivo es el de clarificar el papel de los psicólogos en materia de reclutamiento y Selección, tanto en los departamentos de recursos Humanos como en las consultoras que prestan servicios en esta área.

Para el estudio se seleccionaron 129 empresas, entre las que se encontraban las del iBeX-35 y otras representativas de los diferentes sectores de actividad, así como las consultoras más relevantes. La tasa de respuesta fue del 29,2%

Los resultados indican que la mayoría de los responsables de reclutamiento y selección son psicólogos (65,8%) y que sigue siendo la titulación preferida a la hora de incorporar profe-sionales que se dediquen a esta actividad. La participación de los psicólogos es relevante en todas las fases del proceso de r&S: en la clarificación de la demanda (76,3%), en la elaboración del perfil de exigencias (84,2%), en el reclutamiento y captación de candidatos (89,5%) y en la preselección de candidatos (92,1%). en la fase de evaluación, los psicó-logos son los principales responsables de evaluar inteligencia, aptitudes, personalidad y competencias (81,6%), de la elaboración de informes (89,5%) y participan en la toma de decisiones sobre el candidato elegido (76,3%). Su involucración es algo menor en los se-guimientos de la incorporación (57,9%), desvinculación o salida (28, 9%) y en la auditoria del proceso (44,7%). en cuanto a las técnicas utilizadas, la mayoría (73,7%) utilizan test psicológicos. el póster ofrecerá otras informaciones recogidas en la encuesta.

68—6968—6968—69

¿Cómo puntuar las diferentes opciones en un ítem de un test situacional?.

— Miguel Ángel Sorrel1, Francisco José Abad1, Vicente Ponsoda1, Julio Olea1, Juan Francisco Riesco2, Maria Luisa Vidania2

reSUmen

mientras que en los test de habilidad cognitiva está claro cuál de las opciones a un ítem es la correcta, en los test de juicio situacional a menudo no existe una alternativa objetivamente correcta, de tal manera que varias de las opciones pueden resultar plausibles. esto hace necesario el desarrollo de procedimientos de asignación de puntuaciones. en el presente estudio se han generado dos alternativas a la propuesta original para valorar las respues-tas al Test de integridad de Becker (Becker, 2005). Se tradujo la prueba al castellano y se aplicó a una muestra de 182 personas. Para comprobar la validez convergente y discrimi-nante del test, se aplicaron también medidas de los cinco rasgos básicos de personalidad (neo-FFi), la escala de deseabilidad social de marlowe y Crowne (SDS) y un cuestionario de integridad construido al efecto. Los resultados indican que las dos puntuaciones alter-nativas dan lugar a mejores propiedades psicométricas en lo referido a fiabilidad y validez.

1Universidad autónoma de madrid2Gabinete Psicopedagógico de la academia de Policía Local de la Comunidad de madrid.

70—71

NOTAS

70—71

72—73

72—73

http://www.iic.uam.es/catedras/map

Documents

V Seminario TEST DE JUICIO SITUACIONALcatedras.iic.uam.es/Publicaciones/V_seminario_MAP.pdf · 2-3 • Programación 5 • Prof. Filip Lievens 6-29 • Prof. Fiona Patterson 30-41