Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
Generalitat de CatalunyaDepartament d’Ensenyament
The assessment of speaking and the CEFR
Neus Figueras
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
Th i i d h l d iThe action-oriented approach to language use proposed in the CEFR (2001) reflected, to a considerable extent, theie s of spoken disco rse anal sts hich describedviews of spoken discourse analysts, which described
spoken language as a picture of “dynamism, fluidity,variability mixing and negotiation” (McCarthy:1998)variability, mixing and negotiation (McCarthy:1998).However, and despite the obvious impact of the CEFR in language policies after its publication the changes in thelanguage policies after its publication, the changes in theteaching, the learning and the assessment of the spokenlanguage are yet to be documented. This session willlanguage are yet to be documented. This session willrevise and problematise issues that may not yet have beenaddressed in the field of teaching and assessing speaking,g g p gand will explore possible ways of doing so.
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
The widespread movement to establish educational standards hasl d t k bl i i ff t t l t lled to a remarkable increase in efforts to evaluate learnerperformance through various types of objective testing procedures.At universities it is the CEFR that has primarily set new standardsAt universities, it is the CEFR that has primarily set new standardsin language teaching. In this context, language teachers are beingincreasingly confronted with the task of testing languageincreasingly confronted with the task of testing languageaccomplishments. The 3rd Bremen Symposium 2011 will respond to the demand forThe 3rd Bremen Symposium 2011 will respond to the demand for information, dialogue and exchange on these issues. Thesymposium will be concerned with all aspects surrounding they p p gcentral question of which areas of language competence can bemeasured by which procedures (as put into practice in languageclassrooms), as well as which areas of language competence canbe related to self-imposed or predetermined standards.
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
• Language competenceg g• Learner performance
CEFR Standards
University (context)U e s ty (co te t)
• Assessment constructs• Assessment methods
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
Assessment methods
Generalitat de CatalunyaDepartament d’Ensenyament
L ( ki ) d h CEFRLanguage assessment (speaking) and the CEFR, ten years later. y
Neus Figueras
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
Policy Definitiony
Elaborated Description
Test Content
Cut Score on Test
R k 2008
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
Reckase 2008
Policy Definitiony
Elaborated Description
Test Content
Cut Score on Test
R k 2008
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
Reckase 2008
Revisiting issues
• The nature of speaking*• The nature of speaking• Teaching and learning speakingg g p g• Assessing speaking • Standards and cutscores
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
What is speaking?From Austin (1955) …To say something is in the full normal sense to “do” something,
To Bachman (1990) …language competence (organisational, pragmatic) strategic competence (assessment, planning, execution)
h h i l i l h ipsychophysiological mechanisms
To the CEFR (2001)To the CEFR (2001)…views users and learners of a language primarily as “socialagents” i e members of society who have tasks to accomplishagents , i.e. members of society who have tasks to accomplish (not exclusively target-related) in a given set of circumstances, ina specific environment and within a particular field of action
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
a specific environment and within a particular field of action.
The CEFR action oriented approachThe CEFR action oriented approach
4 1 C f4.1. Context of use4.2. Communication themes4.2. Communication themes4.3. Communicative tasks and purposes4.4. Communicative language activities and
strategies g4.5. Communicative language processes4 6 T t4.6. Texts
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
Oral production Sustained monologuePublic announcementsAddressing audiencesg
Understanding a native speaker interlocutorConversation
Oral interactionInformal discussionFormal discussion and meetingsGoal-oriented cooperationTransactions to obtain goods and servicesInformation exchangeInterviewing and being interviewed
Interaction strategies TurntakingCo-operatingAsking for clarification
Production strategiesAsking for clarification
PlanningCompensating
Mediation* No illustrative scales are yet available
CompensatingMonitoring and repair
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
Mediation No illustrative scales are yet available
Teaching and learning speakingTeaching and learning speaking
(Speaking) is the vehicle par excellence of social solidarity, of social ranking, of professionaladvancement and of business. It is also a mediumthrough which much language is learnt, and which formany is particularly conductive for learning Perhapsmany is particularly conductive for learning. Perhaps,then, the teaching of speaking merits more thought.
Bygate, 1987
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
.. And the car driver analogy:
The job we do when we speak is similar: we do notThe job we do when we speak is similar: we do notmerely know how to assemble sentences in the
b t t h t d th d d tabstract: we have to produce them and adaptthem to the circumstances. This means makingdecisions rapidly, implementing them smoothly, andadjusting our conversation as unexpected problemsadjusting our conversation as unexpected problemsappear in our path.
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
It is the feeling that things are still out of kilter with regard to our ambitions to teach “real language”,g g g ,“language for communication” and “spoken skills” onthe one hand, and our readiness (or otherwise) to( )accept revised descriptions of target languagesbased on what their speakers actually say on thep y yother hand
McCarthy, 1998
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
McCarthy’s proposal
“What should and can be taught in the spoken language”What should and can be taught in the spoken language (1988:2):
The three I’sIllustration InteractionInteraction Induction
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
TensionsLanguage Learning
Language UsingLanguage Using
Teaching
Assessment
Testing
ValidityValidity
Reliability
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
Assessment specificationsAssessment specifications(Luoma, 2004)
Construct specificationConstruct specification
Task specification
Assessment specification
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
• Assessment of learningg• Assessment as learning• Assessment for learning
(Assessment Reform Group, 2002)
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
• Assessment of learningg• Assessment as learning• Assessment for learning
(Assessment Reform Group, 2002)
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
Effective test development(Downing and Haladyna 2006)
1. Overall plan2 Content definitionQ 2. Content definition 3. Test specifications4 It d l t
Q
U4. Item development5. Test design and assemblyA
L 6. Test production7. Test administration
L
I8. Scoring test responses9. Passing scores
I
T10. Reporting test results11. Item banking
Y
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
g12. Test technical report
McNamara’s Model 1996Rater
Rating (score) Marking schemeMarking scheme
Performance
Interlocutor Instrument/Task
Candidate (Underlying Competencies,
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
Skehan)
• A score is a number or a letter that representsA score is a number or a letter that represents how well an examinee perfoms along a continuum.continuum.
A t d d i t t t b t h th• A standard is a statement about whether an examination performance is good enough for a
ti lparticular purpose.-The special score that serves as the boundary p ybetween passing and failing- The numerical answer to the questionThe numerical answer to the question“How much is enough?”
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
77
Pass DistinctionGoodPass Good
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
A2 B2B1A2 B1
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
What is your standard?What is your standard?
How do I know that my B1 is your B1?
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
Policy Definitiony
Elaborated Description
Test Content
Cut Score on Test
R k 2008
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
Reckase 2008
January 2009
Relating Language Examinations to the CommonEuropean Framework of Reference for Languages:European Framework of Reference for Languages:Learning, Teaching, Assessment (CEFR)
A Manual
Language Policy Division, Strasbourg
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
Building an argument
Familiarisation with the CEFRFamiliarisation with the CEFR
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
“Linking of a test to the CEFR cannot be validunless the examination or test that is the subject of the linking can demonstrate validityin its own right”in its own right …
“A test that is not appropriate to context willA test that is not appropriate to context willnot be made more appropriate by linking to the CEFR; “
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
“An examination that has no procedures forAn examination that has no procedures forensuring that standards applied by interviewersor markers are equivalent in severity, or thatsuccessive forms of tests administered insuccessive forms of tests administered indifferent sessions are equivalent,cannot makecredible claims of any linkage of its standard(s)t th CEFR b it t d t tto the CEFR because it cannot demonstrateInternal consistency in the operationalisation ofy pIts standard(s).” (2009:7)
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
Where to start?Where to start?
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
What is Specification? D
• Content analysis of an
What is Specification?T
R
G
O
D
O
C• Content analysis of an examination
R
A
N
O
D
C
U
M• Relating to the CEFR content
and scalesS
PP
M
E
N
• Profiling to CEFR levelsA
R
R
AT
A• Claim of linkageE
N
C
T
I
T
IC
Y
I
C
E
O
N
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
E
“The emphasis in the procedures presented inthi h t li b th d tthis chapter lies on both process and outcome.Users are encouraged to go through a processof content analysing and linking. It is stronglyadvised to reconsider every interim claim thatad sed to eco s de e e y te c a t athas been made during the process. It is quitepossible that the initial estimation of thepossible that the initial estimation of therelationship to the CEFR that was given in FormA8 will need to be revised. The user shouldrevisit the analysis and make a consideredyjudgement. The estimation (Form A8) is confirmedor corrected in Form A24 ” (2009:41)The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
or corrected in Form A24. (2009:41)
What is standard-setting?g“The process of establishing one or more cutscores on examinations. The cut scores divide thedistribution of examinee’s test performances intoptwo or more categories”
“…it is perhaps the branch of psychometrics thatblends more artistic political and cultural ingredientsblends more artistic, political, and cultural ingredientsinto the mix of its products than any other”
Cizek and Bunch 2007:5
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
• Standardisation of judgementsReceptive skillsProductive skillsProductive skills
• Judgement sessionsTasks and itemsScripts and Performances
• Setting cutscores How good is good enough
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
Text characteristicsText characteristicsAbstract/concreteGeneral/specialisedSimple/complex words &structuresL / h tLong/shortReadability measuresTest taker
Topical knowledgeReading experienceStrategic abilities
Response Empirical difficulty
Task
Strategic abilities Test wiseness Text + item demands
Retrieve factual informationMake straightforward
Pilot/test
?JudgeTopical knowledge
ginferencesInterpret, integrate and evaluate meaning and style
Difficulty
?
Reading experienceStrategic abilities+Beliefs
Lexical relations/bondsy
estimationJudgement
Item(s)characteristicsType + NumberGridding
Testing experience
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
McNamara’s model, 1996,Rater
Rating (Score) Scale
PerformancePerformance
Interlocutor Instrument/Task
Candidate (Underlying Competencies, P. Skehan)
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
Stepsp
Wh d H M• Selecting the Panel • Familiarisation
Who and How ManyWhatFamiliarisation
• Selection of approach How
• Completion of procedures• Reporting documentation
How much
• Reporting, documentation and dissemination
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
What is validation?Validation concerns the body of evidence put a da o co ce s e body o e de ce puforward to convince the test users that the wholeprocess and its outcomes are trustworthy.
• Test
• Standard settingStandard setting
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
Standard Setting ValidationStandard Setting Validation
Procedural Explicitness
Standard
Practicability
Standard Setting
ValidationInternal Implementation
Feedback
External Documentation
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011(Cizek & Bunch, 2007:60)
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
Available literature• Council of Europe Manual and Reference supplement
• Cizek, G.J. and M.B.Bunch, Standard Setting. A guide t t bli hi d l ti f t d dto establishing and evaluating performance standards on tests. Sage
• Figueras, N. and Noijons, J. (eds) (2008) Linking to th CEFR L l R h ti C il fthe CEFR Levels: Research perspectives. Council of Europe – CITO. Available on line www.ealta.eu.org
• Martinyuk, W. (ed) (2010) Aligning tests to the CEFR. St di i l t ti CUP
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
Studies in language testing. CUP
Available exemplar items & performancesAvailable exemplar items & performances
• Council of Europe item CD• Council of Europe item CD • Ceftrainhttp://www.helsinki.fi/project/ceftrain/index.php.35.html• Dutch CEFR construct projectp jhttp://www.lancs.ac.uk/fss/projects/grid/• Council of Europe scripts• Council of Europe scriptswww.coe.int• CIEP performanceswww.ciep.fr
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
p
"Spoken performances illustrating the sixlevels of the CEFR“ (www ciep fr)levels of the CEFR (www.ciep.fr)
Examples of oral productions of young people from 13 to 18 years illustratingpeople from 13 to 18 years, illustrating in English, French, German, Italian and Spanish the 6 levels of the CEFR. p
P d t f i i d i 2008Product of a seminar organized in 2008 by CIEP with the assistance of the Language Policy Division of theLanguage Policy Division of the Council of Europe with Cambridge ESOL, Cervantes Institute,
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
Eurocentres Foundation, Goethe Institute, University of Perugia
Conclusions?Conclusions?• A system is available (Manual relevantA system is available (Manual, relevant
literature, toolkit) G id li (EALTA ILTA ALTE )• Guidelines (EALTA, ILTA, ALTE,…)
• Hard work is neededHard work is needed• No easy answers• Systematicity• DocumentationDocumentation• On going process
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
AND, HOPEFULLY, SOME FUN!
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011
Thank you for your attention!
The 3rd Bremen Symposium on Foreign Language Teaching and Learning at Universities – March 4-5, 2011