Embed Size (px)
Citation preview
Test comparability Test comparability across languagesacross languages
Ülle TürkUniversity of Tartu/Estonian Defence
30-31 March 2007 2
Topics to be discussedTopics to be discussed
1. Issues in comparing examinations in different languages (on the basis of the Estonian Year 12 examinations).
2. Some ways of achieving the comparability of examinations in different languages (on the basis of the Finnish matriculation examinations).
3. Relating language examinations to the CEFR – general principles.
4. Relating reading papers to the CEFR. 5. Relating writing papers to the CEFR.
Issues in comparing Issues in comparing examinations in different examinations in different
languageslanguages (on the basis of the Estonian (on the basis of the Estonian
Year 12 examinations)Year 12 examinations)
30-31 March 2007 4
Why compare examinations Why compare examinations across languages?across languages?
• Needs of test users– University admissions officers– Employers– Teachers, students, parents
• Increasing mobility of the population• Increasing consumer choice• A growing emphasis on accountability
30-31 March 2007 5
Equivalence of Equivalence of examinationsexaminations
• Equivalent forms =– Different versions of the same test, which are
regarded as equivalent to each other in that they are based on the same specifications and measure the same competence. To meet the strict requirements of equivalence under classical test theory, different forms of a test must have the same mean difficulty, variance, and co-variance, when administered to the same persons.
ALTE. 1998. Multilingual Glossary of Language Testing Terms.
30-31 March 2007 6
Year 12 examinations in Year 12 examinations in EstoniaEstonia
=national school-leaving examinations established in 1997
–centrally developed, administered and marked–test the learning outcomes of the National Curriculum (2002)
–contain tasks at different levels of difficulty–results on a 100-point scale–offered in 13 subjects
• four foreign languages:English, French, German, Russian
–students required to take three:mother tongue + two more
30-31 March 2007 7
Foreign language Foreign language examinationsexaminations
•National Curriculum for foreign languages– two compulsory foreign languages
• FL A: grades 1–3; 3 hrs per week/ 2 hrs per week• FL B: grades 4–6; 3 hrs per week/ 2 hrs per week
– level B2 in ONE foreign language•Ministry regulations on the development, administration and grading of examinations and reporting results
– five papers: listening and reading comprehension, speaking, writing, language structures
– equally weighted (20 points each)•High-stakes examinations
– results used for university entrance– foreign language requirement: English, French, German
30-31 March 2007 8
QuestionsQuestions•Does 93 in the examination of 2006 reflect the same level of competence as 93 in the examination of 2004 (or 2001)?
•Is 63 in English the same as 63 in German, French or Russian?
•What does ‘the same’ mean?– The same level of competence? B2– The level of competence reached after the same
amount of work?• US Foreign Service Institute (Jackson & Kaplan, 1999: 78)• US Defense Language Institute Foreign Language Center
(MacWhinney 1995: 294)• Threshold Level (Trim)
30-31 March 2007 9
Year 12 examinations in Year 12 examinations in FLsFLs
2001 2004No Mean
%No Mean
%
English 8488 64.1 9099 66.6
German
1408 66.0 1076 66.7
Russian
509 69.4 445 72.3
French 67 75.4 63 79.6
30-31 March 2007 10
German & English: reading German & English: reading paperpaper
• 50 minutes• Three texts with tasks• Length of texts: 1500 words• No of tasks:
– German: one task per text– English: one or two tasks per text
• No of items– German: 20– English: 40
30-31 March 2007 11
Reading: mean scoresReading: mean scores
46%62%84%60%2004
55%67%67%61%2001
Germ
an
57%75%78%68%2004
69%/35%
72%/66%
71%61%2001
En
glis
h
Text 2Text 2Text 1PaperYearLang
30-31 March 2007 12
Text typesText types
German English2001 & 2004
2001 2004
Text 1: Short adverts Newspaper interview
Opinion article
Text 2: Short news/ opinion articles
Short book reviews
Magazine article
Text 3: Newspaper/magazine article (600–700 words)
Newspaper article
Newspaper article
30-31 March 2007 13
Task typesTask typesGerman English
2001 & 2004
2001 2004
Task 1: matching (person + advert)
matching (question + answer)
1) inserting sentences into text 2) multiple choice: 3
Task 2: matching (text + heading)
1) identifying important information2) matching (def + word)
identifying important information
Task 3: multiple choice: 4 options
1) matching (text + title) 2) summary cloze
1) T/F/NI2) matching (def + word)
30-31 March 2007 14
More questionsMore questions
• The reading paper in English seems more difficult than that in German (B2/C1? B1/B2?)
• Why are then the mean scores for the German examination the same or lower than the mean scores for the English examination?– Students who take German are less
motivated and less bright?– It takes longer for Estonian students to
reach the same level of competence in German than in English.
30-31 March 2007 15
Foreign languages in Foreign languages in EstoniaEstonia
2004–5 Percentage of
learners
Examinees: 11033
English 82.4% 9415 85.3%
German 19.2% 1053 9.6%
Russian 39.2% 485 4.4%
French 2.7% 80 0.7%
30-31 March 2007 16
U.S. Government Proficiency RatingsU.S. Government Proficiency Ratings (Jackson & Kaplan, 1999: 73)
Rating Description
S/R-0 No Functional Proficiency
S/R-1 Elementary Proficiency: Able to satisfy routine courtesy and travel needs and to read common signs and simple sentences and phrases.
S/R-2 Limited Working Proficiency: Able to satisfy routine social and limited office needs and to read short typewritten or printed straightforward texts.
S/R-3 General Professional Proficiency: Able to speak accurately and with enough vocabulary to handle social representation and professional discussions within special fields of knowledge; able to read most materials found in daily newspapers.
S/R-4 Advanced Professional Proficiency: Able to speak and read the language fluently and accurately on all levels pertinent to professional needs.
S/R-5 Functionally Equivalent to an Educated Native Speaker
30-31 March 2007 17
Approximate Learning Expectations at Approximate Learning Expectations at the Foreign Service Institutethe Foreign Service Institute
(Jackson & Kaplan, 1999: 78)Language “categories” Weeks Class
hours
Category I: Languages closely cognate with English. French, Italian, Portuguese, Romanian, Spanish, Swedish, Dutch, Norwegian, Afrikaans, etc.
23-24 575-600
Category II: Languages with significant linguistic and/or cultural differences from English. Albanian, Azerbaijani, Bulgarian, Finnish, Greek, Hebrew, Hindi, Hungarian, Icelandic, Khmer, Latvian, Nepali, Polish, Russian, Serbian, Tagalog, Thai, Turkish, Urdu, Vietnamese, Zulu, etc.
44 1100
Category III: Languages which are exceptionally difficult for native English speakers to learn to speak and/or read. Arabic, Chinese, Japanese, Korean
88(2nd year
in the country)
2200
30-31 March 2007 18
The Defense Language Institute Foreign The Defense Language Institute Foreign
Language CenterLanguage Center (MacWhinney 1995: 294)
Group I Western European languages: Roman alphabet, share many cognates with English, greatly simplified grammatical system.
Group II More challenging Indo-European languages: Roman alphabet, complex grammatical system (Lithuanian, German, Romanian, Hindi)
Group III
Indo-European languages: non-Roman writing system, complex grammar (Greek, Russian, Serbian, and Persian)“Easy” non-Indo-European languages (Hungarian, Tagalog, Turkish; Thai or Vietnamese)
Group IV Non-Indo-European languages: non-Roman writing system, complex grammar (Arabic, Japanese, and Korean)
Group V Exotic languages: Eskimo, Warlpiri, Navajo, Georgian, etc.
Some ways of achieving the Some ways of achieving the comparability of comparability of
examinations in different examinations in different languages languages
(on the basis of the Finnish (on the basis of the Finnish matriculation examinations)matriculation examinations)
30-31 March 2007 20
Preparing examinationsPreparing examinations
• Joint examination board for foreign languages (16 people)
• 2–3 members representing each language• Language groups – 5–6 people• Markers (for English: 20–30 people)
• Collective responsibility: the board as a whole is responsible for the quality of all language examinations
30-31 March 2007 21
Process of examination Process of examination preparationpreparation
• Language groups– Agree on text and task types– Agree on the schedule of work– Find the texts (2–3 times as many as will be needed)– Design the materials
• The board discusses all the examination papers– Constructive criticism– Suggest changes/improvements
• Language groups make the necessary changes• All the members of the language group read the
test materials to make sure that they contain no mistakes
30-31 March 2007 22
Post-examination analysisPost-examination analysis
• 3–4 weeks for marking the papers• Teachers mark their own students’ papers first
using very detailed marking schemes, but their marks do not count.
• If the central marker’s grade differs from that given by the teacher too greatly, a second marker is brought in.
• The whole board analyses the examination results• If a question does not ‘work’, all students are
awarded a point for it.• Item difficulty is taken into consideration when
awarding the grades.
30-31 March 2007 23
GradingGrading
• 7 grades based on norm referencing– Laudatur (5%)– Eximia (10%)– Magna cum laude (20%)– Cum laude (30%)– Lubenter approbatur (20%)– Approbatur (10%)– Improbatur (2-5%)
• reliability – 0.9-0.95
Relating language Relating language examinations to the CEFR – examinations to the CEFR –
general principlesgeneral principles
30-31 March 2007 25
Relating an examination or test to the CEF is a complex endeavour. The existence of such a relation is not a simple observable fact, but is an assertion for which examination provider needs to provide both theoretical and empirical evidence. The procedures by which such evidence is put forward can be summarized by the term “validation of the claim.”
Relating Language Examinations to the CEF: Manual (Preliminary Pilot Version), 2003: 1
30-31 March 2007 26
ProceduresProcedures (1) (1)• Familiarisation:
– A selection of activities designed to ensure that participants in the linking process have a detailed knowledge of the CEFR
• Specification:– A self-audit of the coverage of the examination
(content and task types) in relation to the categories presented in CEFR Chapters 4 (Language use and the language learner) and 5 (The user/learner’s competences)
30-31 March 2007 27
ProceduresProcedures (2) (2)
• Standardisation:– Suggested procedures to facilitate the
implementation of a common understanding of the Common Reference Levels presented in CEFR Chapter 3
• Empirical validation:– The collection and analysis of test data and
ratings from assessments in order to provide evidence that both the examination itself and the linking to the CEFR are sound
30-31 March 2007 28
Specification of examination Specification of examination contentcontent (1) (1)
• Familiarisation with CEFR– Consideration of a selection of the
question boxes printed at the end of relevant sections of CEFR chapters
– Discussion of the CEFR levels as a whole– Self-assessment of own language level in
a foreign language– Sorting individual CEFR descriptors into
levels
30-31 March 2007 29
• Internal validity: Description and analysis of– general examination content– process of test development– marking, grading, results– test analysis and post-examination review
• External validity: Relate– general examination description to CEFR scales– description of communicative activities tested to
CEFR scales– description of aspects of communicative
language competence tested to CEFR scales
Specification of examination Specification of examination contentcontent (2) (2)
30-31 March 2007 30
Standardisation of Standardisation of judgementsjudgements (1) (1)
• Familiarisation with CEFR as in the Specification stage (2 h)
• Productive skills– Training in assessing performance in
relation to CEFR levels using standardised samples (3–4 h/skill)
– Benchmarking local performance samples to CEFR levels (3–4 h/skill)
30-31 March 2007 31
Standardisation of Standardisation of judgementsjudgements (2) (2)
• Receptive skills– Training in judging the difficulty of
test items in relation to CEFR standardised items (3–4 h/skill)
– Judging the difficulty of local items in relation to CEFR levels (3–4 h/skill)
30-31 March 2007 32
Empirical validationEmpirical validation
• Data collection• Internal validation:
– Confirming the psychometric quality of the test
• External validation:– Confirming the relationship to the
CEFR through an independent measure
30-31 March 2007 33
ActivitiesActivities 1 1• Familiarisation
– Consideration of a selection of the question boxes printed at the end of relevant sections of CEF chapters 3, 4 and 5
– Discussion of the CEFR levels as a whole• Table 1. Common Reference Levels: global scale (p
24)
– Sorting individual CEFR descriptors into levels• Spoken Fluency (p 129)• General Linguistic Range (p 110)
30-31 March 2007 34
3 Common Reference Levels3 Common Reference Levels• Users of the Framework may wish to consider and
where appropriate state:– to what extent their interest in levels relates to learning
objectives, syllabus content, teacher guidelines and continuous assessment tasks (constructor-oriented);
– to what extent their interest in levels relates to increasing consistency of assessment by providing defined criteria for degree of skill (assessor-oriented);
– to what extent their interest in levels relates to reporting results to employers, other educational sectors, parents and learners themselves (user-oriented), providing defined criteria for degrees of skill (assessor-oriented);
– to what extent their interest in levels relates to reporting results to employers, other educational sectors, parents and learners themselves (user-oriented).
30-31 March 2007 35
4 Language use and the 4 Language use and the language user/learnerlanguage user/learner
• Users of the Framework may wish to consider and where appropriate state:– in which domains the learner will need/be
equipped/be required to operate.• Users of the Framework may wish to
consider and where appropriate state:– the situations which the learner will
need/be equipped/be required to handle;– the locations, institutions/organisations,
persons, objects, events and actions with which the learner will be concerned.
30-31 March 2007 36
5 The user/learner’s 5 The user/learner’s competencescompetences
• Users of the Framework may wish to consider and where appropriate state:– what prior sociocultural experience and knowledge the learner
is assumed/required to have;– what new experience and knowledge of social life in his/her
community as well as in the target community the learner will need to acquire in order to meet the requirements of L2 communication;
– what awareness of the relation between home and target cultures the learner will need so as to develop an appropriate intercultural competence.
• Users of the Framework may wish to consider and where appropriate state:– on which theory of grammar they have based their work;– which grammatical elements, categories, classes, structures,
processes and relations are learners, etc. equipped/required to handle.
30-31 March 2007 37
Can understand the main points of clear standard input on familiar matters regularly encountered in work, school, leisure, etc. Can deal with most situations likely to arise whilst travelling in an area where the language is spoken. Can produce simple connected text on topics which are familiar or of personal interest. Can describe experiences and events, dreams, hopes & ambitions and briefly give reasons and explanations for opinions and plans.
Can understand and use familiar everyday expressions and very basic phrases aimed at the satisfaction of needs of a concrete type. Can introduce him/herself and others and can ask and answer questions about personal details such as where he/she lives, people he/she knows and things he/she has. Can interact in a simple way provided the other person talks slowly and clearly and is prepared to help.
Can understand with ease virtually everything heard or read. Can summarise information from different spoken and written sources, reconstructing arguments and accounts in a coherent presentation. Can express him/herself spontaneously, very fluently and precisely, differentiating finer shades of meaning even in more complex situations.
B1
A1
C2
30-31 March 2007 38
Profi-cient user
C2 Can understand with ease virtually everything heard or read. Can summarise information from different spoken and written sources, reconstructing arguments and accounts in a coherent presentation. Can express him/herself spontaneously, very fluently and precisely, differentiating finer shades of meaning even in more complex situations.
C1 Can understand a wide range of demanding, longer texts, and recognise implicit meaning. Can express him/herself fluently and spontaneously without much obvious searching for expressions. Can use language flexibly and effectively for social, academic and professional purposes. Can produce clear, well-structured, detailed text on complex subjects, showing controlled use of organisational patterns, connectors and cohesive devices.
30-31 March 2007 39
Inde-pen-dentUser
B2 Can understand the main ideas of complex text on both concrete and abstract topics, including technical discussions in his/her field of specialisation. Can interact with a degree of fluency and spontaneity that makes regular interaction with native speakers quite possible without strain for either party. Can produce clear, detailed text on a wide range of subjects and explain a viewpoint on a topical issue giving the advantages and disadvantages of various options.
B1 Can understand the main points of clear standard input on familiar matters regularly encountered in work, school, leisure, etc. Can deal with most situations likely to arise whilst travelling in an area where the language is spoken. Can produce simple connected text on topics which are familiar or of personal interest. Can describe experiences and events, dreams, hopes & ambitions and briefly give reasons and explanations for opinions and plans.
30-31 March 2007 40
BasicUser
A2 Can understand sentences and frequently used expressions related to areas of most immediate relevance (e.g. very basic personal and family information, shopping, local geography, employment). Can communicate in simple and routine tasks requiring a simple and direct exchange of information on familiar and routine matters. Can describe in simple terms aspects of his/her background, immediate environment and matters in areas of immediate need.
A1 Can understand and use familiar everyday expressions and very basic phrases aimed at the satisfaction of needs of a concrete type. Can introduce him/herself and others and can ask and answer questions about personal details such as where he/she lives, people he/she knows and things he/she has. Can interact in a simple way provided the other person talks slowly and clearly and is prepared to help.
30-31 March 2007 41
Can express him/herself at length with a natural, effortless, unhesitating flow.
Can keep going comprehensibly, even though pausing for grammatical and lexical planning and repair is very evident, especially in longer stretches of free production.
Can construct phrases on familiar topics with sufficient ease to handle short exchanges, despite very noticeable hesitation and false starts.
Can interact with a degree of fluency and spontaneity that makes regular interaction with native speakers quite possible without imposing strain on either party.
Can manage very short, isolated, mainly pre-packaged utterances, with much pausing to search for expressions, to articulate less familiar words, and to repair communication.
Can make him/herself understood in short contributions, even though pauses, false starts and reformulation are very evident.
Can communicate spontaneously, often showing remarkable fluency and ease of expression in even longer complex stretches of speech.
Can produce stretches of language with a fairly even tempo; although he/she can be hesitant as he/she searches for patterns and expressions, there are few noticeably long pauses.
C2
B1A2
B2
A1
B1C1
B2
30-31 March 2007 42
Has a very basic range of simple expressions about personal details and needs of a concrete type.Has a sufficient range of language to be able to give clear descriptions, express viewpoints and develop arguments without much conspicuous searching for words, using some complex sentence forms to do so.Can select an appropriate formulation from a broad range of language to express him/herself clearly, without having to restrict what he/she wants to say.Has enough language to get by, with sufficient vocabulary to express him/herself with some hesitation and circumlocutions on topics such as family, hobbies and interests, work, travel, and current events.Can use basic sentence patterns and communicate with memorised phrases, groups of a few words and formulae about themselves and other people, what they do, places, possessions etc. Can exploit a comprehensive and reliable mastery of a very wide range of language to formulate thoughts precisely, give emphasis, differentiate and eliminate ambiguity.Can produce brief everyday expressions in order to satisfy simple needs of a concrete type: personal details, daily routines, wants and needs, requests for information.Has a sufficient range of language to describe unpredictable situations, explain the main points in an idea or problem with reasonable precision and express thoughts on abstract or cultural topics such as music and films.
A1
B2C1B1A2C2A2
B2
30-31 March 2007 43
SourcesSources•Common European Framework of Reference for Modern Languages: Learning, Teaching, Assessment (CEFR): http://www.coe.int/T/DG4/Portfolio/?L=E&M=/documents_intro/common_framework.html
•Hardcastle, Peter. 2004. Test Equivalence and Construct Compatibility across Languages. University of Cambridge ESOL Examinations Research Notes, 17, August, 6−11.
•Jackson, Frederick H. & Kaplan, Marsha A. 1999. Lessons learned from fifty years of theory and practice in government language teaching. In: Georgetown University Round Table on Language and Linguistics. Washington, DC: Georgetown University Press, 71–87.
30-31 March 2007 44
SourcesSources
• MacWhinney, Brian. 1995. Language-Specific Prediction in Foreign Language Learning. Language Testing, 12, 292–320.
• Manual for relating language examinations to the Common European Framework of Reference for Languages – a preliminary pilot version: http://www.coe.int/T/DG4/Portfolio/?L=E&M=/documents_intro/Manual.html
• Taylor, Lynda. 2004. Issues of test comparability. University of Cambridge ESOL Examinations Research Notes, 15, February, 2–5.
