Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus NussbaumGilles Guillard, Markus Nussbaum

The European Union

Economic &Social Committee

Committee of the Regions

Ombudsman

Data ProtectionSupervisor

Council

Parliament

Court of Justice

Court of Auditors

Commission

The EU Institutions

Posts in 2012

European Parliament 6655

Council of the European Union 3153

European Commission 25478

Court of Justice 1952

European Court of Auditors 887

European Economic and Social Committee 724

Committee of the Regions 531

European Ombudsman 66

European Data Protection Supervisor 43

European External Action Service 1670

Total 41159

European External

Action Service4% Others

3%

European Parliament

16%

Council of the

European Union

8%

European Commission

62%

European Court

of Auditors2%

Court of Justice

5%

Administrators (AD)e.g. policy officers, lawyers, auditors, scientific officers, translators, interpreters, communication & press officers …

Qualification: University degree (bachelor)

Assistants (AST)e.g. secretaries, HR assistants, financial assistants, conference technicians …

Qualification: Secondary education (at least) and relevant professional experience

Administrators (AD)e.g. policy officers, lawyers, auditors, scientific officers, translators, interpreters, communication & press officers …

Qualification: University degree (bachelor)

Assistants (AST)e.g. secretaries, HR assistants, financial assistants, conference technicians …

Qualification: Secondary education (at least) and relevant professional experience

Entry Streams

Selection in Cycles

Competition launched in March 2012

45000+ candidates for 296 posts

5 fields: Audit, EU Public Administration, Communication, Law, External Relations

Competition launched in March 2012

45000+ candidates for 296 posts

5 fields: Audit, EU Public Administration, Communication, Law, External Relations

The AD Cycle 2012

Verbal reasoning20 items from a database, 35 min, 23 languages

Numerical reasoning10 Items from a database, 20 min, 23 languages

Abstract reasoning10 Items in fixed test forms, 10 min

Situational Judgement20 Items in fixed test forms, 30 min, EN/DE/FR

Verbal reasoning20 items from a database, 35 min, 23 languages

Numerical reasoning10 Items from a database, 20 min, 23 languages

Abstract reasoning10 Items in fixed test forms, 10 min

Situational Judgement20 Items in fixed test forms, 30 min, EN/DE/FR

Computer Based Tests

Verbal Reasoning

Numerical Reasoning

Abstract Reasoning

X

X

Situational Judgement

Results

EU Public Admin Audit Communication Law External Rel

Sum

AD5 AD7 AD5 AD7 AD5 AD7 AD5 AD7 AD5 AD7

validated application 12287 7892 2675 1389 3467 2023 4636 2130 6519 2338 45356

booked CBT 9557 6076 2182 1005 2525 1430 3615 1592 5016 1706 34704

participated CBT 8323 5336 1942 893 2132 1263 3113 1330 4412 1470 30214

reached CBT pass-marks 5502 3767 1427 603 1253 723 2076 830 2935 948 20064

succeeded CBT 190 102 108 34 77 51 144 43 64 21 834

Score Distribution - VR

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

0250500750

10001250150017502000225025002750300032503500

3750

AD177: VR

participated

admitted

reserve list

Test score

Co

un

t

0 1 2 3 4 5 6 7 8 9 10

0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

5500

6000

AD177: NR

participated

admitted

reserve list

Test score

Co

un

t

Score Distribution - NR

0 1 2 3 4 5 6 7 8 9 10

0500

1000150020002500300035004000450050005500600065007000

7500

AD177: AR

participated

admitted

reserve list

Test score

Co

un

t

Score Distribution - AR

Situational Judgement

AD-2011: SJ-AP

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Score

Co

un

t

AD-2011: SJ-RE

0

1000

2000

3000

4000

5000

6000

7000

8000

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Score

Co

un

t

AD-2011: SJ-QR

0

1000

2000

3000

4000

5000

6000

7000

8000

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Score

Co

un

t

AD-2011: SJ-PO

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Score

Co

un

t

AD-2011: SJ-WO

0

1000

2000

3000

4000

5000

6000

7000

8000

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Score

Co

un

t

Intercorrelations

Verbal R

easoning

Num

erical Reasoning

Abstract R

easoning

SJT

Analysis &

Problem

S

olving

SJT

Planning &

Organising

SJT

Quality &

Results

SJT

Resilience

SJT

Working w

ith others

Verbal Reasoning - .41 .29 .16 .23 .15 .17 .18

Numerical Reasoning .41 - .32 .15 .20 .14 .15 .15

Abstract Reasoning .29 .32 - .14 .17 .12 .14 .14

SJT Analysis & Problem Solving .16 .15 .14 - .40 .33 .31 .35

SJT Planning & Organising .23 .20 .17 .40 - .32 .35 .35

SJT Quality & Results .15 .14 .12 .33 .32 - .35 .33

SJT Resilience .17 .15 .14 .31 .35 .35 - .32

SJT Working with others .18 .15 .14 .35 .35 .33 .32 -

How EPSO monitors the behaviour of items

How EPSO monitors the behaviour of items

Item Analysis

Based on the dicotomous / polytomous Rasch model

Conducted after every majorcompetition

Calibrate items and monitor their behaviour

Making psychometrics as vivid as possible

Based on the dicotomous / polytomous Rasch model

Conducted after every majorcompetition

Calibrate items and monitor their behaviour

Making psychometrics as vivid as possible

Item Analysis

Georg Rasch (1901-1980)

Test Responses Candidates Items

VR 2 295 360 114 768 5 071

NR 1 147 680 114 768 3 490

Total 3 443 040 8 561

Example:Cumulative Analysis VR+NR

Oct 2012

Item Difficulty Adjustment Across Languages (IDAAL)

How EPSO ensures fair opportunities for all candidates although testing in 23 different languages

How EPSO ensures fair opportunities for all candidates although testing in 23 different languages

Aligned Averages Algorithm (AAA)

A

0

1

2

3

4

-4

-3

-2

-1

5

EN

0

1

2

3

4

-4

-3

-2

-1

5

EN

A

-4-3

-2-1

0+

1+

2+

3+

4

Aligned Averages Algorithm (AAA)

citem = common item in en and lang

lang

m

encitemencitem

n

langcitemlangcitem

mn

1)(

)(1)(

)(

Minimum Meansquare Method (MMM)

0

1

2

3

4

-4

-3

-2

-1

5

EN A B C

A

B

C

0

1

2

3

4

-4

-3

-2

-1

5

EN

Minimum Meansquare Method (MMM)

min!

2

1 1

1,

,

n

item

m

lang

m

langlanglangitem

langlangitem m

Person Parameter Contrast Measure (PPCM)

0

1

2

3

4

-4

-3

-2

-1

5

Cdts EN

Cdts A

Person Parameter Contrast Measure (PPCM)

lang

m

encdtencdt

n

langcdtlangcdt

mn

1)(

)(1)(

)(

... is a combination of these three approaches:

Aligned Averages Algorithm (AAA)

Minimum Meansquare Method (MMM)

Person Parameter Contrast Measure (PPCM)

... is a combination of these three approaches:

Aligned Averages Algorithm (AAA)

Minimum Meansquare Method (MMM)

Person Parameter Contrast Measure (PPCM)

Item Difficulty Adjustment Across Languages (IDAAL)

EPSO Rasch Item Combination Algorithm (ERICA)

EPSO Rasch Item Combination Algorithm (ERICA)

How to assemble test forms

Candidates have to be treated equally: Same number of items Same difficulty of test form

➾ Adaptive testing is not possible

EPSO has developed an algorithm involving two steps:

1. Set up a "master test form"

2. Create test forms corresponding to master test form

Candidates have to be treated equally: Same number of items Same difficulty of test form

➾ Adaptive testing is not possible

EPSO has developed an algorithm involving two steps:

1. Set up a "master test form"

2. Create test forms corresponding to master test form

Framework

Actual test forms correspond to master test form

Limited overlap between test forms

No over-exposed items

Further improving gender balance

Actual test forms correspond to master test form

Limited overlap between test forms

No over-exposed items

Further improving gender balance

Target (quality criteria)

Defining the master test form

easy difficult

Master test form difficulty: 8

Master test form difficulty: 2

Master test form difficulty: 5

Items are approximately standard normal distributed

Most items are around the average (0) available

Weighting was implemented to better take account of item availabilities

Items are approximately standard normal distributed

Most items are around the average (0) available

Weighting was implemented to better take account of item availabilities

Weighting

Sample Test FormTarget Item Actual

-1.00 EN1626VEN -1.001

-0.56 EN1555V -0.550

-0.22 EN1170V -0.227

0.06 EN1063V 0.062

0.30 EN2361V 0.295

0.51 EN1003VEN 0.517

0.71 EN1686V 0.703

0.89 EN2089V 0.891

1.06 EN2041VEN 1.064

1.22 EN2133V 1.211

1.38 EN2191V 1.394

1.54 EN2261V 1.532

1.69 EN2260VEN 1.684

1.84 EN2180VEN 1.811

2.00 EN2196VEN 2.032

2.16 EN2264V 2.198

2.33 EN2142V 2.333

2.51 EN1578VEN 2.451

2.73 EN1292VEN 2.809

3.00 EN2371V 2.941

0.9Sum SqDev:

Thank you very much for your attention!

Any questions? Any questions?