ELT J-1999-Foot-36-41

7/29/2019 ELT J-1999-Foot-36-41

1/16

Relaxing in pairsMichael C. FootThis article looks at paired tests of spoken language. It regrets the lack ofpublished research evidence, and of results from the monitoring of thesetests to support their introduction and wider use . It questions some of theclaims fo r these tests, and wonders whether they are really more effective,or efficient than one-to-one tests.It is customary for writers on tests of spok en lang uage (H ea ton 1975: 83,Madsen 1983: 147) to begin with the observation that while speaking isthe most important of the language skills, it is also the most difficult totest. To the familiar problems of validity and reliability are added thedifficulties caused by most tests of spoken language being 'live' tests,requiring the presence of the examiner, and the related considerations ofcost and efficiency. The value of some recent innovations in tests ofspoken language is not apparent, and in the writer 's view may only haveadded to the difficulties.One of the major innovations is the practice of pairs of candidates being._- .-. tested by pairs of exam iners. This pro ced ure is considere d to have a_number of advantages, in particular that of helping candidates to feelrelaxed, o r 'psychologically ea sier' (W allis 1995). Th e p resen ce of twoexaminers, i t is argued, reassures candidates, because it removes theirfear of individual examiner bias, and ensures greater marker reliabili ty.The format is said to have other advantages, for example, that i tprovides the opportunity for a more varied pattern of conversation thanone-to-one tests.

The candidates Attempting to make tests less of an ordeal for candidates is a worthyaim. It is disappointing, however, that although the University ofCambridge Local Examinat ions Syndicate has ten years ' experience ofexam ining candid ates in pairs at one level, neithe r their original resea rchfindings, nor the results of any subsequent monitoring of the changeshave been published. Where candidates had the choice of being testedalone or in pairs, many still opted for the one-to-one test, but it isdifficult to gauge candidates' actual preferences, because someexamination centres had a policy of encouraging pairs-only testing.If research revealed a marked preference by candidates for being testedin pairs, then clearly it would be necessary to review the on e-to-o ne test .But the fact that some candidates find being tested in pairs moreenjoyable, or easier, is not sufficient reas on to ab an do n o ne-to -one tests.It would also be necessary to demonstrate that the performance ofcandidates tested in pairs is better than, or at least as good as, thosetested singly.

36 ELT Journal Volume 53/1 January 1999 O xford University Press 1999

atUniversityofCoventryonDecember9,2012

http://eltj.oxfordjournals.org/

Downloade

dfrom
http://eltj.oxfordjournals.org/http://eltj.oxfordjournals.org/http://eltj.oxfordjournals.org/http://eltj.oxfordjournals.org/http://eltj.oxfordjournals.org/http://eltj.oxfordjournals.org/http://eltj.oxfordjournals.org/http://eltj.oxfordjournals.org/http://eltj.oxfordjournals.org/http://eltj.oxfordjournals.org/http://eltj.oxfordjournals.org/http://eltj.oxfordjournals.org/http://eltj.oxfordjournals.org/http://eltj.oxfordjournals.org/http://eltj.oxfordjournals.org/http://eltj.oxfordjournals.org/http://eltj.oxfordjournals.org/http://eltj.oxfordjournals.org/http://eltj.oxfordjournals.org/http://eltj.oxfordjournals.org/http://eltj.oxfordjournals.org/http://eltj.oxfordjournals.org/http://eltj.oxfordjournals.org/http://eltj.oxfordjournals.org/http://eltj.oxfordjournals.org/http://eltj.oxfordjournals.org/http://eltj.oxfordjournals.org/http://eltj.oxfordjournals.org/http://eltj.oxfordjournals.org/http://eltj.oxfordjournals.org/http://eltj.oxfordjournals.org/http://eltj.oxfordjournals.org/http://eltj.oxfordjournals.org/http://eltj.oxfordjournals.org/

7/29/2019 ELT J-1999-Foot-36-41

2/16

It is argued (Wallis 1995) that candidates will be relaxed if they knoweach other, and have things in common to talk about. So, typically,paired tests begin with the candidates, in turn, telling one of theexaminers (the interlocutor) about the other. The theory, presumably, isthat any anxiety felt by the candidates is reduced by being shared.However, the relationship between stress and performance is not asimple one; many people do not perform better because they arerelaxed. Also, it is not an uncommon experience to find that otherpeople's nervousness is catching rather than reassuring.Surprisingly, it is maintained (Wallis 1995) that it does no t matter if thetwo candidates do not know each other, or have anything in common.This is because the situation is thought to create a natural informationgap which mimics real-life conversation. The difficulty here is inequating the creation of a relaxed atmosphere with the creation of anatural information gap; of telling the examiner about a friend withinterviewing a stranger. The consequence is that there are actually twodifferent tests; one if the candidate is a friend, and a different one if theother candidate is a stranger.Testing in twos appears to be based on the assumption, or hope, that thecandidates will share the same first language. If they do not, then, asevery EFL teacher knows, the result can come close to mutualincomprehensibility. Most examiners have difficulty, at some time, inguessing what a candidate means. Is it reasonable to ask a candidate,particularly one at the lower levels of proficiency, to decipher another'sidiosyncratic syntax and pronunciation, where even the examiner hasproblems? Given the likelihood that many candidates will be unfamiliarwith the accent of candidates with a different mother tongue, any testwhich requires candidates to engage in conversation with each other isclearly biased in favour of candidates who share the same mothertongue.Of course, examiners have instructions to make allowances for acandidate who, reasonably, has difficulty understanding the othercandidate. How precisely does an examiner make an allowance? Whatis the level or frequency of difficulty that triggers a compensatoryassessment? How is it calculated? How does the final assessment reflectthe fact that one candidate was disadvantaged by an incomprehensibleor uncomprehending partner? Surely the frustrated candidate in suchcircumstances has a right to know.The truth is that it is impossible to treat a candidate fairly in thesecircumstances. The examiner is likely to find it difficult to adhere to thestandard procedure, or to obtain the adequate sample of languagenecessary to make a proper assessment. Where one candidate isconsistently unable to return service, the most experienced examiner cannever be sure exactly how the better candidate would have performedwith a different partner, or on their own. Any assurance that 'allowancewill be made' is a forlorn one.Relaxing in pairs 37

atUniversityofCoventryonDecemb

er9,2012


Downloade

dfrom

7/29/2019 ELT J-1999-Foot-36-41

3/16

Examiners are instructed not to penalise candidates if they havedifficulty understanding each other, and to encourage them to seekclarification rathe r than remain silent. The need to issue such guidance isan admission of the real constraints under which some candidates arelikely to be placed in paired tests. Worse, the guidance reveals very littleunderstanding of the candidate's situation. Is it likely that a candidatepaired with a friend would wish to do anything to draw attention to thefriend's lack of proficiency, and diminish the friend's chances of success?On the contrary, some stronger candidates might hold back, and notperform to their full potential, in the belief that they are helping theirfriends. In fact, it is pretty clear that on occasions this is precisely whathappens. And what cause would candidates have to behave anydifferently to strangers?From the writer's observations, there is a feeling that many teachers areconcerned about how differences in the ability of paired candidates canaffect performance. If examining bodies' research on pairs testingsuggests that differences in the ability of candidates do not affectperformance, it would be reassuring to see it. There is a quite widely-held view, reinforced by daily experience, that where the level ofparticipants in an activity is markedly different, the performance of eachis affected. For example, it is a common experience of many tennisplayers to play better against a good opponent than against an inferiorone. It would be surprising, to find that something similar does nothappen in pairs testing.Jones (1985: 81) has suggested that 'ease of elicitation in an oralinterview is often affected by the social relationship between theexaminer and examinee. The factors can include age, race, social class,and profession.' W here clear differences in age and status exist betweencandidates in paired tests, they appear equally likely to affect thecandidates' composure and performance. In most societies, to pair ayoung with a mature student would be to display a lack of considerationfor both. That these arrangements are countenanced is a surprisingdeparture from the basic humanism for which the paired approach totesting appears to stand.There is also the question of personality. Candidates are warned againstattempting to dominate their partners. However, nothing as crude asdomination is required to put the other candidate at a disadvantage. Inpaired tests candidates are often asked to discuss a problem together,and to try to reach agreem ent on it. We can imagine a situation in whichCandidate A, a slightly reserved person, has formulated an opinion andis ready to express it, but Candidate B , without being over-assertive, getsin first and makes the same pointa not infrequent occurrence innatural conversation. Candidate A will be unlikely to get much credit, ifany, for repeating what Candidate B said, no matter how much betterexpressed. In a one-to-one test, Candidate A would have been assessedvery differently. Thus, a procedure adopted to produce a more relaxedatmosphere could also subtly draw attention to the fact that the two

38 Michael C. Foot


er9,2012


Downloadedfrom

7/29/2019 ELT J-1999-Foot-36-41

4/16

The examiners

Different patternsof exchange

candidates are actually in competition with each other.As well as eliminating candidates' concern about possible individualexaminer bias, the use of two examiners is intended to relieve thepressure under which single examiners work in the trad itional interview-based test. The single examiner is replaced by an interlocutor and anassessor.Typically the interlocutor explains the tasks to the candidates, engagesthem in conversation during the introductory stage of the test, asks themto explain their solution to any joint task, and acts as time-keeper. Theassessor listens to the candidates and assesses them on the evidence oftheir performance in the tasks, against the established criteria.Assessors may, towards the end of the test, talk to the candidates for thepurpose of 'fine tuning' the assessment. However, if an assessor needsadditional evidence from one candidate to confirm a provisionalassessment, considerations of standardization and validity require theassessor to waste time questioning the other candidate as well!Another drawback with a two-examiners approach is that it threatens,even more obviously than the singleton interview, the illusion of anatural conversation. An assessor who takes little or no part in the testbut simply observes, patently offends the norms of natu ral conversation,and reinforces in the candidates' minds the essentially contrived natureof the event.Whether we are writing or speaking, we are always addressing someone.The best person to assess whether that communication is effective is theperson addressed, and effective communication includes non-verbal aswell as verbal communication. W here one of the exam iners plays no partin the exchange, that examiner must always be an observer. It isaxiomatic that observers invariably have a different perspective fromparticipants, whatever the activity. Thus, the interlocutor and assessormay disagree in their assessments, not because they apply the criteriadifferently, but because they actually see and hear different things; oneis a participant, the other a spectator. The notion, therefore, that in thesecircumstances two examiners ensure greater marker reliability, is notconvincing.Another of the alleged advantages of the pairs format is that it givescandidates a better chance to show what they can do. Different patternsof exchange, candidate with interlocutor, candidate with candidate, andcandidates with interlocutor, provide opportunities to employ a widerrange of speech events than the one-to-one interview. But, argument,description, discussion, narrative, and opinion are not speech eventspeculiar to three-way conversations and, if they observe their instruc-tions, the interlocutors' interventions will be minimal, and most of thetest will, anyway, consist of a dialogue between the candidates.

Relaxing in pairs 39



Downloade

dfrom

7/29/2019 ELT J-1999-Foot-36-41

5/16

In fact, unless the candidates a re well-matched, their a ttempts to sustaina discussion are likely to be, and often are, faltering and desultory, andthe outcom e, for them a sense of frustration rather than of achievem ent.Although this is unlikely to be the experience of candidates at higherlevels of proficiency, at lower levels it is not uncommon.It is difficult to see how a discussion between two inexpert users,struggling to overcome the ir own limitations, and a ttempting to deciphe rthe opacities of the other, is compatible with providing candidates withthe optimum conditions for showing what they can do.In such circumstances, instructions to the interlocutors not to intervenein the discussion between the candidates appear perverse. If candidatesare allowed to flounder with unfamiliar topics, or to dominate theconversation, who is being helped to show what they can do? What hashappened to the examiner's responsibility to obtain an adequate sampleof language? Without examiner control, the amount candidates talk willvary considerably one from another, as in authentic conversation. Theexaminer does not have unlimited time to obtain the varied sample oflanguage required to make an assessment. Unless the examiner canintervene and redirect the discussion, there is a risk the assessment willreflect the paucity of data rather than the candidates' abilities.

Leng th of tests One of the surprising things about testing in twos is not only how littletime the examiners engage the candidates in conversation, but how littletime the candidates themselves spend talking. I take for illustrationUCLES Certificate of Advanced English (CAE). During the first twophases of the CAE (the introductions and the information gap task), thecandidates engage in three monologues. If the candidates do not knoweach other, the pattern is an interview and two monologues. The testtimetable requires each candidate to speak for about one minute in thefirst phase , and one m inute 20 seconds in the second . Th e last two phasesof the test are the problem-solving task (a discussion between the twocandidates), and an open discussion involving both candidates and theinterlocutor and, possibly, the assessor. After making allowance for theinterlocu tor's explanation of the task, giving the candidates a moment tocollect their thoughts, and time for the interlocutor's and the assessor'squestions, the maximum time available for each candidate to talk islikely to be abo ut 3 minutes 30 seconds. Thus, in an interview lasting 15minutes, each candidate will speak for about 5 minutes 50 seconds. Tomany it will come as a surprise that 5 minutes 50 seconds is consideredsufficient time to assess the language proficiency of advanced-levelcandidates.It is probable that many will feel as Hughes (1989: 105) that, 'It isunlikely that much reliable information can be obtained in less than 15minutes, while 30 minutes can provide all the information necessary formost purposes.' Is it really the case that the quality of the data obtainedby two examiners in five or six minutes is likely to be superior to thatobtained by one examiner in a one-to-one test lasting fifteen minutes?

40 Michael C. Foot



Downloade

dfrom

7/29/2019 ELT J-1999-Foot-36-41

6/16

On the grounds that the more one sees of a candidate the morerepresentative the sample is likely to be, Hughes' view would certainlybe endorsed by most experienced recruitment and selection inter-viewers.The discrepancy between the length of the examination and the amountof time the candidates actually spend talking is largely taken up with themanagement of the different tasks which comprise the test. To enablethe candidates to perform the tasks, the interlocutor must first explainthem, and then introduce the materials to be used in each. The m aterialscan be sets of photographs or drawings or diagrams. In any informationgap exercise, each candidate has to have a different set. The interlocutoris a sort of master of ceremonies who is required to present each task,and make sure the right candidate has the right materials. Clearly theneed for two examiners is, as much a consequence of the complexities ofsome paired tests, as a concern for marker reliability, and theelimination of examiner bias.

Conclusion Undoubtedly some candidates will find paired tests less stressful thansingleton interviews. However, before removing the option of one-to-one tests, we ought to be sure that paired tests do create the optimumconditions for all candidates. At the moment it looks as though inaddressing one problem, the examiners have created several new ones.Some aspects of paired tests clearly have the potential to be unfair. It isalso debateable whether this technique, which may work well at higherlevels of proficiency, is appropriate at lower levels. Finally, there appearto be grounds for wondering whether paired tests really are a moreefficient way of examining spoken language than one-to-one tests.

ReferencesHeaton, J. B. 1975. Writing English LanguageTests. London: Longman.Hughes, A. 1989. Tests for Langu age Teachers.Cambridge: Cambridge University Press.Jones, R. L. 1985. 'Some Basic Considerations inTesting Oral Proficiency' in Lee Y. P. et al.(eds). New Directions in Language Testing:papers presented at the International Sy mposiumon Language Testing. Hon g Kong, 1985. Oxford:Pergamon Press.

Madsen, H. S. 1983. Techniques in Testing. NewYork: Oxford University Press.Wallis, D. 1995. 'Testing in twosthe pairedformat in oral examinations'. UCLES presenta-tion, IATEFL Annual Conference, York.

The authorMichael Foot is a freelance EFL teacher, anexaminer of spoken English, and a former Directorof Personnel in the British Council. He has an MA inELT from the University of Essex.

Relaxing in pairs 41



Downloade

dfrom

7/29/2019 ELT J-1999-Foot-36-41

7/16

Assessing speaking in therevised FCENick Saville and Peter HargreavesThis paper describes the Speaking Test which forms part of the revisedFirst Certificate of English (FCE) examination produced by the University ofCambridge Local Examinations Syndicate (UCL ES), and introduced for thefirst time in December 1996 (see First Certificate in English: Handbook,UCLES, 1997J. The aim is to present the new test as the outcome of arational process of test development, and to consider why the new designprovides improvements in the assessment of speaking within the FC Econtext.While examinations by their nature tend to be conservative, theCambridge examinat ions produced over the years have kept pace withchanges in English teaching, so that modifications to the examinationshave taken place in an evolutionary way. FCE, first introduced in 1939under the ti t le Lower Certificate in English, has been revisedperiodically over the years in order to keep pace with changes inlanguage teaching and language use, and also as part of an ongoingcommitment to test validation. Prior to the revision introduced in 1996,FCE underwent major revisions in 1984, and before that in 1973. Bychanging in this way, it has been possible to continue to achieve positiveimpact in the contexts where the examinations are usedespecially inrelation to English language learning, and teaching around the world.In this respect, one of the key features of UCLES EFL examinations hasbeen a focus on the assessment of speaking by means of a face-to-facespeaking test as an obligatory component of the examinations. As partof the revisions to FCE and CPE, in order to keep up with developmentsin the field, UCLES has introduced new procedures and a number ofdifferent speaking-test formats have been used. Across the range ofexaminat ions that are now produced by UCLES, there is current ly nosingle model for testing speaking. Some examinations, l ike theInternat ional English Language Test ing System (IELTS), employ aspeaking test in the one-to-one format (i .e. with one candidate and oneexaminer), and all tests are recorded so that they can be rated by otherexaminers at a later stage. Other examinations make use of a groupformat in the speaking tests, with more than two candidates assessedtogether (as in the Certificate in English for English LanguageT e a c h e r s C E E L T ) .

Elicitation and In designing a face-to-face speaking test such as those employed byratings UCLES, the test developer has to produce a sui table procedure whichinvolves two main aspects:

42 ELT Journal Volume 53/1 January 1999 Oxford University Press 1999


er9,2012


Downloade

dfrom

7/29/2019 ELT J-1999-Foot-36-41

8/16

a) the elicitation of an appropriate sample of spoken English;b) the rating of that sample in terms of pre-defined descriptions ofperformance in spoken English, whether as a whole, or brokendown into different criteria (e.g. accuracy, range, pronunciation,etc.).These aspects in turn depend on two factors:the availability of valid and reliable materials and criterion ratingscales;the development and support of a professional oral examiner cadre.In designing a speaking test, there are no right or wrong solutions to thisproblem; as Bachman and Palmer (1996) point out, an appropriate('useful') outcome is achieved by balancing the essential qualities ofvalidity, reliability, impact, and practicality to meet the requirements ofthe testing context. Despite the variety of formats which are used fortesting speaking, in relation to the following examinations which formthe Cambridge 5-Level System, UCLES has taken steps in recent yearstowards harmonization of approach:

Cambridge Level 5Certificate of Proficiency in EnglishCPECambridge Level 4Certificate in Advanced EnglishCAECambridge Level 3First Certificate in EnglishFCECambridge Level 2Preliminary English TestPETCambridge Level 1Key English TestKETThe aim is to establish common features which can be appliedappropriately at the different levels. Some of the more importantfeatures which have been identified in this process have beenincorporated into the revision of the FCE, and can be summarized asfollows:

a) A paired format, based on two candidates and two oral examiners.b) Of the two oral exam iners, one acts as interlocutor, and his or hermost important role is to manage the discourse (i.e. ensure that anappropriate sample is elicited from each of the paired candidates);

the other acts as assessor, and is not involved in the interaction.c) There are different phases or parts to the FCE Speaking Test,which facilitate the assessment of different patterns of interaction,participant roles, discourse, rhetorical functions, etc.d) Standard ization of formats is achieved partly by the use ofcontrolled interlocutor frames, and partly by the use of tasks basedon visual stimuli from generic sets appropriate to the level andnature of the examination.e) Both the interlocutor and the assessor rate the candidates'

performance, but the interlocutor provides a global/holisticassessment, while the assessor provides an analytical assessment.With the revision of FCE, there are now four examinations which makeuse of the paired formatKET, PET, FCE, and CAE (CPE is currentlyAssessing speaking in the revised FCE 43



Downloade

dfrom

7/29/2019 ELT J-1999-Foot-36-41

9/16

under review, and st i l l retains the option of the one-to-one approach).The paired format Th e decision to use the paired format as the standard mo del for the m ainsuite speaking tests has been a key feature in balancing the essential testquali t ies in relation to the contexts where these examinations are used.

The paired test was first used with FCE and CPE as an optional formatduring the 1980s. W hen C A E was introduced in 1991, the paired formatwas established as an obligatory feature of one of the main suite tests forthe first t ime . This was extende d to K ET in 1993, to the revised P E T in1995, and most recently to the revised FCE in 1996. Before the decisionwas made to extend the use of the paired format across the range ofexaminations, and especially to FCE, the various alternative formatswere evaluated. In part icular, feedback was collected from a wide rangeof 's takeholders ' in the tests from around the world (including oralexaminers, teachers, students, and candidates taking the tests). Inaddit ion, a range of validation projects carried out by the UCLES EFLDivision in the 1990s have co ntributed to a greater un dersta ndin g of thiskind of assessment procedure (e.g. Lazaraton 1996a, 1996b, Milanovic,Saville, Pollitt, and Cook 1996, Young and Milanovic 1992).One of the major advantages of the paired format is the use of twoexaminers to assess a candidate. This adds to the fairness of theassessment, and helps to make candidates feel reassured that their markdoes not just depend on one person. The paired format also allows morevaried patterns of interaction during the examination; whereas in theone-to-one model there is only one interaction pattern possible (i .e.in teract ion between one exam iner and one candidate) , the pai red formatprovides the potential for various interaction patterns between eachcandidate and the examiner, and between the candidates themselves. Inaddit ion, the paired format has the potential for posit ive washback, inencouraging more interaction between learners in the classroom.Any test format presents test developers with a range of potentialproblems and issues which need to be addressed, and the paired formatof the speaking tests is no exception. Critics of the paired format areoften concerned with issues relating to the pairing of the candidates. It isargued, for example, that the paired format may not provide eachcandidate with an equal opportunity to perform to the best of theirability, or that the pairings may influence the assessment (e.g. due tomismatch of language levela good candidate paired with a weakeroneor if one candidate is paired with another of a different age,gender, or nationali ty).Many of these concerns cannot be addressed with definit ive answers.However, the potential problems need to be seen within the context ofthe overall design of the examination, and balanced against the posit iveadvantages . Fo r examp le, U C LE S has at temp ted to address the issue ofhow much spoken language is produced by each candidate:

a) by paying close attention to the design of the different parts of thetests;44 Nick Saville and Peter Hargreaves


er9,2012


Downloade

dfrom

7/29/2019 ELT J-1999-Foot-36-41

10/16

Features of therevised FCESpeaking Test

b) by providing examiners with an interlocutor frame to follow whilstadministering the examination.These features, together with comprehensive training for oral examiners(described below), help to ensure that a balanced sample of speech iselicited, and that each candidate receives an equal opportunity toperform to the best of his or her ability during the examination.Moreover, the EFL Division at UCLES has been conducting researchsince 1992 on the discourse produced in paired-format speaking tests,and work specifically related to the speech of candidates in FCE hasbeen going on since 1995. The purpose of this research is to betterunderstand the features of the language produced during a pairedformat test. Initial findings related to the revised FCE suggest that thefeatures of candidate language predicted by the test specifications werepresent in the samples which were analysed. In this regard, it isimportant to understand how the format of the revised FCE SpeakingTest was arrived at, and the steps taken to ensure that the assessment isstandardized. The second half of this paper describes in more detail thefeatures of the revised FCE Speaking Test, and the way that oralexaminers are trained and co-ordinated to carry out the test procedures,and to make appropriate ratings.The initial context for the most recent revision of FCE was provided bythe existing uses of the examination, and the nature of the currentcandidature. What was already known of existing standardstheexpected level of performance by FCE candidates (centred on passingcandidates with a grade C)provided the background for the revisedassessment criteria, and the application of the rating scales. The ratingscales themselves (for use by oral examiners) were redeveloped inrelation to the harmonized approach to the assessment of speaking,described above.Within this approach, all criteria used in the assessment were defined,and related to a model of Communicative Language Ability (CLA). Thecriteria and scales for revised FCE are derived from the same model

Figure 1 Spoken language abilityLanguage competence Strategic competence

Pragmaticrammatical Discourse

SyntaxMorphologyVocabularyPronunciationRhetoricalorganisationCoherenceCohesion

e.g.Sensitivityto illocutionInteraction skillsNon-verbal featuresof interaction

Assessing speaking in the revised FCE 45


er9,2012


Downloade

dfrom

7/29/2019 ELT J-1999-Foot-36-41

11/16

which underpins the revision project as a whole, based on thedevelopments in this area during the 1980s, e.g. the work of Canaleand Swain (1980), Bachman (1990), and the Council of Europespecifications for Waystage and Threshold (1990). See Figure 1.The revised FCE has five assessment criteria in all, four analytical andone global: gram mar and vocabulary, discourse man agem ent , pron un-ciation, interactive communication, and global achievement.

Test format and In the revised FCE, the Speaking Test consists of four parts, each oftask features which focuses on a different type of interaction: betw een the interlocu torand each candidate, between the two candidates , and among al l three.The patterns of discourse vary within each part of the test , andcandidates are encouraged to prepare for the Speaking Test bypractising talking individually, and in small groups with the teacherand with peers. The aim is to help them to be aware of, and to practise,the norms of turn-taking, and the appropriate ways of participating in aconversation, or taking up a topic under discussion. This is seen as oneaspect of positive impact that the test can achieve.Oral examiners make use of a task features specification whichsummarizes the features of the task which are appropriate to the level,and purpose of the examination. Each part of the test is a separate task,with the following specific features:interaction pattern (examiner to candidate(s); candidate to candidate,etc.), input (verbal and/or visual), and output by candidates.The e xpec ted output of the candidates is predicted from the c om bina-tion of feature s for each task , and is jud ged in relatio n to the irperformance in these tasks, which have been designed according to thelevel of FCE, and in order to provide an appropriate level of difficultyfor the typical FCE candidature. As noted above, the tasks includedifferent interaction patterns, different discourse types (short turn, longturn, etc.), and have features such as turn-taking, collaborating,initiating/responding, and exchanging information. Examples of othertask features include functions such as describing and comparing, statingand supporting an opinion, agreeing and disagreeing, speculating,expressing certainty and uncertainty. This is summarized in Table 1.Each task has its own focus:Par t 1InterviewThe interlocutor directs the conversation, by asking each candidate togive some basic personal information about him or herself. T h ecandidates do not need to talk to each other in this part of the test ,though they may if they wish.Part 2Long turnEach candidate is given the opportunity to talk without interruption onhis or her own for about one minute. Each candidate is asked tocompare and contrast two colour photographs, commenting on the

46 Nick Saville an d Peter Hargreaves


er9,2012


Downloadedfrom

7/29/2019 ELT J-1999-Foot-36-41

12/16

Table 1: Task features Parts

1 Interview(3 minutes)

2 Individuallong turn(4 minutes)

3 Two-waycollaborativetask(3 minutes)

4 Three-waydiscussion(4 minutes)

Task formatInteractionpatterninterlocutorinterviewscandidates

interlocutordelegates anindividual taskto eachcandidate

interlocutordelegates acollaborativetask to the pairof candidates

interlocutorleads adiscussion withthe twocandidates

Inputverbalquestions

visual stimuliwith verbalrubrics

visual/writtenstimuli, withverbal rubrics

verbal prompts

Candidate outputDiscoursefeaturesresponding toquestionsexpanding onresponses

sustaining along turnmanagingdiscourse:coherenceand clarity ofmessageorganizationof languageand ideasaccuracy andappropriacy oflinguisticresourcesturn-taking:initiating andrespondingappropriatelynegotiating

initiating andrespondingappropriatelydevelopingtopics

Functionsgiving personalinformationtalking aboutpresentcircumstancestalking aboutpastexperiencetalking aboutfuture plansgivinginformationexpressingopinions, e.g.throughcomparing andcontrastingexplaining andgiving reasons

exchanginginformationand opinionsexpressing andjustifyingopinionsagreeing and/or disagreeingsuggestingspeculatingexchanginginformationand opinionsexpressing andjustifyingopinionsagreeing and/or disagreeing

pictures, and giving some personal reaction to them. They are notrequired to describe the photographs in detail.Part 3Two-way collaborative taskThe candidates are provided with a visual stimulus (one or severalphotographs/line drawings/computer graphics, etc.) to form the basis fora task which they attempt together. Sometimes the candidates may beasked to agree on a decision or conclusion, whereas at other times theymay be told that they may agree to disagree. In all cases, it is theAssessing speaking in the revised FCE 47


er9,2012


Downloade

dfrom

7/29/2019 ELT J-1999-Foot-36-41

13/16

working towards the completion of the task that counts, rather than theactual completion of the task.Par t AThree-way discussionThe interlocutor again directs the conversation by encouraging thecandidates to bro ade n and discuss further th e topics introduced in Part 3.In the information about the test which is provided to candidates, i t ismade clear that they must be prepared to provide full but naturalanswers to questions asked either by the interlocutor or the othercand idate, and to speak clearly and audibly. They should not be afraid toask for clarification if they have not understood what has been said. Ifmisunderstandings arise during the test , candidates should ask theinterlocutor, or each other, to explain further. Obviously, no marks aregained by remaining silent, and equally, no marks are lost for seekingclarification on what is required. On the contrary, this is an importantfeature of strategic ability, which is one of the criteria for assessment(under the interactive communication scale).While i t is the role of the interlocutor, where necessary, to manage ordirect the interaction, ensuring that both candidates are given an equalopportunity to speak, it is also the responsibility of the candidates tomaintain the interaction as much as possible. Candidates who are able tobalance their turns in the interchange will utilize to best effect theamount of t ime available, and so provide the oral examiners with anadequate amount of language to assess.From the point of view of ratings, an advantage of the paired format isthat two independent ratings are obtained for each candidate, thusmaking the examination fairer. In the revised FCE both the assessor andinterlocutor record marks using the same criteria, although the twoexaminers are expected to have slightly different perspectives on theperformance due to their different rolesthe interlocutor as participant,and the assessor as observer. To reflect this, the revised FCE makes useof two types of rating scale: a set of analytical scales derived from thecriteria in the model of Communicative Language Ability, and a globalscale which combines the criteria in the analytical scales in anappropriate way. This rating procedure involves the assessor markingeach candidate on the four analytical scales as the test is in progress, andat the end, the interlocutor giving a single global score for eachcandidate based on the global achievement scale. There is norequirement for the examiners to discuss and agree the marks, andthe final assessment is derived from the two ratings when the marksheets are returned to Cambridge.

Examiner training Successful elicitation and accurate ratings are to a large extentdependent on the knowledge and abili ty of the oral examiners. In thefirst instance, careful test design can help to ensure that the examinersare likely to find the elicitation procedures and rating scales easy toapply. However, the successful functioning of a speaking test, such asthat used in the revised FCE and most other Cambridge examinat ions,

48 Nick Saville an d Peter Hargreaves


er9,2012


Downloadedfrom

7/29/2019 ELT J-1999-Foot-36-41

14/16

relies heavily on a system for training and standardizing the oralexaminers. For UCLES this is a major undertaking, as there arecurrently about 7,000 approved UC LES E FL oral examiners around theworld involved in conducting one or more of the Speaking Tests for theCambridge EFL examinations. The major objectives in regard to theperformance of these oral examiners are that:a) they consistently apply the Speaking Test procedures to obtainrepresentative, valid samples of the candidates' spoken English inaccordance with the test specifications;b) they rate the samples of spoken English accurately and consistently,in terms of the pre-defined descriptions of performance, using therating scales provided by UCLES.Over the years UCLES has developed a two-pronged approach toensuring these objectives can be met, based, firstly, on a network ofprofessionals with various levels of (overlapping) responsibility, and,secondly, on a set of procedures which apply to each professional level.In the network of professionals there are three levels, in addition toUCLES' own staff. At the operational level there are the oralexaminers. At the next level up, in countries where there are sufficientnumbers of oral examiners to merit it, team leaders are engaged by localsecretaries with the responsibility of professional supervision of oralexaminers, in a ratio of about one team leader to between five and 30oral examiners, depending on such factors as distribution of oralexaminers, location of centres, etc. Finally, in countries where thenumber of team leaders (and hence oral examiners) merit it, senior teamleaders have been appointed by UCLES to supervise team leaders in anaverage ratio of one senior team leader to 15 team leaders. This forms ahierarchy of responsibilities. See Figure 2.The levels in this hierarchy are not sealed off from each other: it is arequirement that team leaders and senior team leaders must also bepractising oral examiners, in order to ensure that they can draw on theirexperience when it comes to dealing with the concerns of oralexaminers.The set of procedures which regulate the activities of these threeprofessional levels is summarized by the acronym R-I-T-C-M-E, wherethe initials stand for:

Figure 2 UCLE S

Senior team leaders

Team leaders

Oral examinersAssessing speaking in the revised FCE 49



Downloade

dfrom

7/29/2019 ELT J-1999-Foot-36-41

15/16

R e c r u i t m e n t , I n d u c t i o n , T r a i n i n g , C o - o r d i n a t i o n , M o n i t o r i n g , a n dE v a l u a t i o n .E a c h of th e s e p r o c e d u r e s i s d e f in e d b y a l is t o f M in im u m Pr o f e s s io n a lR e q u i r e m e n t s ( M P R s ) a p p r o p r i a t e t o t h e l e v el o f p r o f e s s i o n a lr e s p o n s i b i l i t y . T h e s e M P R s s e t d o w n t h e m i n i m u m l e v e l s a n d s t a n d a r d s( f o r r e c r u i t m e n t , i n d u c t i o n p r o g r a m m e s , e t c . ) w h i c h m u s t b e a c h i e v e d i no r d e r t o m e e t t h e p r o f e s si o n a l r e q u i r e m e n t s o f a d m i n i s t e r i n g C a m -b r id g e E F L Sp e a k in g Te s t s , a n d to s u s ta in a f ul ly e ff e c tiv e t e a m le a d e rs y s t e m .T h e f ir st t w o p r o c e d u r e s c o v e r e d b y R - I - T - C - M - E , r e c r u i t m e n t a n din d u c t io n , ty p ic a l ly a p p ly o n ly o n c e to a n a p p l i c a n t o r a l e x a min e r f o r ag i v e n e x a m i n a t i o n . T h e r e m a i n d e r o f t h e p r o c e d u r e s a r e r e c u r r e n t , a n dto s o me e x te n t c y c l i c a l f o r e a c h e x a min a t io n , in s o f a r a s th e o u tc o me o fm o n i t o r i n g a n d e v a l u a t i o n f e e d s i n t o t r a i n i n g a n d c o - o r d i n a t i o n .Af te r in i t i a l t r a in in g o f e x a min e r s , s t a n d a r d iz a t io n o f a s s e s s me n t i sm a i n t a i n e d b y t h e a n n u a l e x a m i n e r c o - o r d i n a t i o n s e s s i o n s o f o r a le x a m i n e r s a p p r o v e d f o r r e l e v a n t e x a m i n a t i o n , a n d b y m o n i t o r i n g v is it st o c e n t r e s b y t e a m l e a d e r s . D u r i n g c o - o r d i n a t i o n s e s s i o n s , e x a m i n e r sw a t c h a n d d i s c u ss s a m p l e s p e a k i n g t e s t s r e c o r d e d o n v i d e o , a n d t h e nc o n d u c t p r a c t i c e t e s t s w i th v o lu n te e r ' c a n d id a te s ' i n o r d e r to e s ta b l i s h ac o m m o n s t a n d a r d o f a s s e s s m e n t . T h e s a m p l e t e s t s o n v i d e o a r e s e l e c t e db y UCLES to d e mo n s t r a te a r a n g e o f t a s k ty p e s a n d d i f f e r e n t l e v e l s o f

~~ c o m p e t e n c e , a n d a r e p r e - m a r k e d b y a t e a m o f e x p e r i e n c e d a s s e s s o r s . I nt h i s c o n t e x t , m o n i t o r i n g a n d e v a l u a t i o n r e f e r t o b o t h t h e te s t p r o c e d u r e s( e .g . w h e t h e r t h e p r o c e d u r e e l ic i ts a n a p p r o p r i a t e s a m p l e ) , a n d t o t h ep e r f o r m a n c e o f t h e o r a l e x a m i n e r s . T h i s l a t t e r k i n d o f m o n i t o r i n g a n de v a l u a t i o n f o r m s p a r t o f t h e h u m a n r e s o u r c e a p p r a i s a l s y s t e m , w h i c h i sn e c e s s a r y t o g u a r a n t e e t h e q u a l i t y o f t h e a s s e s s m e n t s . D u r i n g m o n i t o r -i n g , t e a m l e a d e r s c o m p l e t e e v a l u a t i o n s h e e t s f o r t h e o r a l e x a m i n e r sb e i n g m o n i t o r e d ; t h e y d i s c u s s r e s u l t s w i t h t h e o r a l e x a m i n e r s t h e m -s e lv e s , a n d wi th th e lo c a l s e c r e ta r i e s a s p a r t o f a p la n n in g / r e v ie wm e e t i n g . T h e e v a l u a t i o n s h e e t s a r e t h e n s e n t t o t h e s e n i o r t e a m l e a d e r s ,a n d fina lly o n to Ca m b r i d g e fo r a n a ly s i s . I n a d d i t io n , g r e a te r u s e is n o wb e i n g m a d e o f a u d i o re c o r d i n g s t o m o n i t o r b o t h c a n d i d a t e o u t p u t a n de x a m i n e r p e r f o r m a n c e .

Conclusion T h i s p a p e r h a s d e s c r i b e d t h e S p e a k i n g T e s t in t h e r e v i s e d F C E i nr e la t io n to th e f o r ma t o f th e t e s t , a n d th e wa y in wh ic h a s s e s s me n ts a r ema d e , f o c u s in g in p a r t i c u la r o n th e r o le o f th e o r a l e x a min e r s . Wh i le n ote s t a c h ie v e s a p e r f e c t b a la n c e o f n e c e s s a r y q u a l i t i e s , i t i s b e l i e v e d th a tth e c u r r e n t b a la n c e , w i th th e r e c e n t r e v i s io n s , t a k e s s e v e r a l s t e p sf o r w a r d i n t e r m s o f i m p r o v e m e n t s o v e r e a r l i e r s o l u t i o n s . A n o n - g o i n gc o m m i t m e n t t o v a l i d a t i o n in v o l v i n g d a t a c o l l e c t i o n , m o n i t o r i n g , a n de v a l u a t i o n w i l l e n s u r e t h a t t h e e v o l u t i o n a r y p r o c e s s o f c h a n g e c o n t i n u e sin f u tu r e . I n th i s wa y , a n d a s o u r k n o w le d g e o f th e c o m p le x i t i e s o fs p o k e n l a n g u a g e g r o w s , f u r t h e r r e v i s i o n s c a n b e e x p e c t e d i n t h e f u t u r e .

50 Nick Saville and Peter Hargreaves


er9,2012


Downloadedfrom

7/29/2019 ELT J-1999-Foot-36-41

16/16

ReferencesBachman, L. F. 1990. Fundamental Considerationsin Language Testing. Oxford: Oxford UniversityPress.Bachman, L. F. and A. S. Palmer. 1996. LanguageTesting in Practice. Oxford: Oxford UniversityPress.Canale, M. an d M. Swain. 1980. 'The oretical basesof communicative approaches to second lan-guage teaching and testing'. Applied Linguistics1/1:1-47. Oxford: Oxford University Press.van Ek, J. A. and J. L. M. Trim. 1990. ThresholdLevel 1990. Strasbourg: Council of Europe.van Ek, J. A. an d J. L. M. Trim. 1990. Waystage1990. Strasbourg: Council of Europe.Lazaraton, A . 1996a. 'Interlocutor support in oralproficiency interviews: The case of CASE'.Language Testing 13: 151-72. London: Edw ardArnold.Lazaraton. A. 1996b. 'A qualitative approach tomonitoring examiner conduct in the CambridgeAssessment of Spoken English (CASE)' inStudies in Language Testing 3: Performancetesting, cognition and assessment: Selected papersfrom the 15th Language Testing ResearchColloquium (LTRC): 18-33. Cambridge: Cam-bridge University Press/UCLES.Milanovic, M. N., A. Saville, A. Pollitt, an d A .Cook. 1996. 'Developing Rating Scales forCASE: Theoretical Concerns and Analyses' inValidation in Language Testing. Clevedon:Multilingual Matters.

University of Cambridge Local ExaminationsSyndicate. 1997. First Certificate in English-Handbook. Cambridge: University of Cam-bridge Local Examinations Syndicate.Young, R. an d M. Milanovic 1992. 'Discoursevariation in oral proficiency interviews' inStudies in Second Language Acquisition 14:403-24. Cambridge: Cambridge UniversityPress.

The authorsNick Saville has been Group Manager for TestDevelopment and Validation within the EFL Divi-sion of the University of Cambridge Local Examina-tions Syndicate (UCLES) since 1994. His ownresearch interest is in the development and valida-tion of procedures for oral assessment, and he was amember of the UCLES development team thatworked on the revision of the FCE Speaking Test.E-mail: Peter Hargreaves joined UCLES as Director of theEFL Division after working for the British Councilfor over twenty years. His early background in EL Twas in teacher training, but he moved into testingwith the Council after ob taining his doctorate in thediscipline gramm ar of English. He now heads a teamof about 70 staff at UCLES, working on theCambridge EFL examinations and IntegratedLanguage Teaching Schemes.

Assessing speaking in the revised FCE 51



Downloadedfrom

Documents

ELT J-1999-Foot-36-41