Project Goals

© Stichting CITO Instituut voor Toetsontwikkeling 1

Mapping the Dutch Foreign Language State Examinations onto the Common

European Framework of Reference

Report of a Cito research project commissioned by

the Dutch Ministry of Education, Culture and Science

José Noijons & Henk Kuijper


Project Goals

• To establish links between the existing examinations in French, German and English and the CEFR, following the steps as outlined in the Manual published by the Council of Europe.

• To study the possibilities of developing more comprehensive CEFR-related examinations in the foreign languages.


Four Project Phases

Phase 1: Familiarisation

Phase 2: Specification

Phase 3: Standardisation

Phase 4: Empirical validation



Specification of texts

• Related to the reading scales for communicative language activities (CEFR chapter 4)

• Related to text type, source, topic, domain (Dutch Grid)

• Related to the scales for communicative language competence (CEFR chapter 5 & Dutch Grid)



Specification of items• Question types

• Task dimensions 1 (recognising, inferences, evaluation)

• Task dimensions 2 (explicit vs. implicit)

• Task dimensions 3 (content of operations)


Specification texts: reading scales communicative activities German

5

21

173

3

0

50

100

150

200

readingcorrespondence

reading for orientation reading for informationand argument

reading instructions


Specification texts: text type, source, topic, domain

• Great variation in text source and communicative themes (topics)

• Increase of proportion of expository and argumentative texts from lower to higher educational levels

• Domain: mainly personal


Specification texts: communicative language competences

• Level of abstraction• Grammatical complexity• Vocabulary• Text length

Dutch grid


Specification texts: communicative language competences, level of abstraction

0

0,5

1

1,5

2

2,5

3

3,5

4

vmbo-bb vmbo-kb vmbo-gl/tl havo vwo

English

German

Only concrete 1

Mostly concrete 2

Fairly extensive abstract 3

Mainly abstract 4


Specification texts: communicative language competences, grammar

0

0,5

1

1,5

2

2,5

3

3,5

4

vmbo-bb vmbo-kb vmbo-gl/tl havo vwo

English

German

Only simple structures 1

Mainly simple structures 2

Limited range of complex structures 3

Wide range of complex structures 4


Specification items

From lower to higher educational level:• more inferences and evaluation, less

recognising• Relatively more implicit information• Greater variation of operations


Specification: general conclusions

• Increase of linguistic and cognitive complexity of texts

• Increase of variation of operations demanded in the items


Claims through Specification and Standardisation

If a claim of a link to the CEFR is based on specification only, we do not know what score a candidate needs to claim that the candidate’s ability is at the CEFR-level the test claims to be at.

Claims can be further substantiated through standardisation of judgements: item-difficulties are judged in relation to CEFR levels.


Standardisation

Standard-setting

Basket procedure: judges are to assign items (texts + tasks) to CEFR-levels.

What minimum CEFR level does one need to master this item?


Results for Exams in English

Required minimum CEFR level for Exams in English


Results for Exams in French

Required minimum CEFR level for Exams in French


Results for Exams in German

Required minimum CEFR level for Exams in German


Scores & Standards: English

Distribution of scores & cut score vwo-exam English

VWO English

Score distributionScore distribution


Scores & Standards: French

Distribution of scores & cut score vwo-exam French

VWO French

Score distribution

VWO French


Scores & Standards: German

Distribution of scores & cut score vwo-exam German

VWO English

Score distributionScore distribution

VWO German


Conclusions standard-setting

• In the Netherlands a candidate can “pass” an exam without attaining the relevant CEFR level for that exam.

• Only scores at, or higher than the CEFR cut score indicate that the candidate is at or above the CEFR level the exam has been estimated to be at.

• Empirical validation may help to show that a CEFR level (a score) attained on one exam is equivalent to a CEFR level on another exam.


General Conclusions

• A test that is linked or validated through specification only, cannot provide sufficient information on how candidates need to perform on the test to claim they have reached relevant attainment targets.

• It is necessary through standard setting to compute minimum scores that are needed for candidates to claim they have reached relevant attainment targets.

• External validation is needed to verify claims of links to the CEFR.

Documents

Project Goals