Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
Computer-aided approach to language testing: How can computer-aided technology help increase authenticity in language tests?
Dr Dana Gablasova
Lancaster University
Language testing: Authenticity
Authenticity – directly related to the validity and usefulness of a test; at the core of authenticity – the relationship between the characteristics of the test task and the TLU task
“We consider authenticity to be an important test quality because it relates the test task to the domain of generalization to which we want our score interpretations to generalize” (Bachman & Palmer, 1996, p. 23-24)
“Another reason for considering authenticity to be important is because of its potential effect on test takers’ perceptions of the test and, hence, on their performance.[…] It is this relevance, as perceived by the test taker, that we believe helps promote a positive affective response to the test task and can thus help test takers perform at their best”. (Bachman & Palmer, 1996, p. 24).
How can computer-aided technology help increase authenticity in language tests?
CL & LT: Growing cooperation
Corpora and computer-aided analysis – a steady presence in the field of LT for several decades since 1990s (e.g. Alderson, 1996)
Major examination bodies commissioned and used corpora in creation or revision of their tests (e.g. Pearson, ETS, Trinity College London, the British Council, Cambridge English Language Assessment).
The topic of corpus approaches to LTA has increasingly appeared in major reference works in the fields of applied linguistics and language testing (e.g. Routledge Handbooks)
Research articles on corpus-based LTA analysis are on the increase in major
LT journals - Language Testing, Language Assessment Quarterly (e.g. Paquot, 2018; Cushing, 2017; Park, 2014)
AL journals - Applied Linguistics, the Modern Language Journal, Language Learning (Staples et al, 2017, 2018; Wisniewski, 2017)
What are corpora?
Corpora are large electronic collections of actual language use, used as evidence in empirical, (mostly) quantitative analysis of language
Corpora are representative
corpora can represent different modes and domains of language use (e.g. spoken, written, online communication, different domains – engineering, business communication, academic writing)
corpora can represent different groups of language users (e.g. of different gender, age, social status, L2 speakers of different proficiency levels, from different L2 backgrounds)
corpora are searchable according to these characteristics
Example: The British Academic Spoken English (BAWE) corpus
• proficient university-level writing – 6.5M words & almost 3,000 good-standard student assignments
• four disciplinary areas (Arts & Humanities, Social Sciences, Life Sciences and Physical Sciences)
Corpus methods
Corpora are typically large (millions & billions of words)
Corpus methods allow us to Process large amounts of data and manipulate them (sort, search,
filter, calculate statistics)
Annotate the data for linguistic, text-related or speaker-related information(e.g. grammatical structures such as all nouns, all premodified nouns; semantic information, e.g. all words related to emotions); search for language from younger or older speakers…
Identify patterns (regularities) in language use o identify features typical (recurring) in certain texts o relate the patterns to underlying (explanatory) variables (speakers’
proficiency or L1 background, previous instruction, text and task properties)
Corpus methods
Concordance analysis Frequency wordlists
Ellis & Simpson-Vlach, 2010
Collocation analysis Keyword analysis
Corpus evidence complements linguistic intuition
See McEnery and Wilson, 2001: 8
Expert + Native speaker
Corpus information (BNC)
damage death
disruption explosion failure
havoc grievous bodily harm
Corpora reveal (hidden) patterns
Corpora: Two types relevant to LT
Two types of corpora relevant for increasing authenticity in language testing:
1. Corpora representing target language use
2. Corpora representing L2 production under exam conditions
Corpora: Target language use (TLU)
L1 USER & EXPERT CORPORA
A. General corpora – typically very large – seek to represent a broadly defined domain (e.g. all of spoken British English) – samples from different genres/registers from this domain
Example: The British National Corpus 2014 • Built in collaboration btw Lancaster University & CUP • Contains a written (ongoing) and spoken part (completed) • Covers different genres of written British English (fiction,
academic prose, journalistic prose, poetry, drama, student essays, emails, twitter messages, …) + informal conversations (10M words)
Corpora: Target language use (TLU)
B. Specialised corpora – represent more narrowly defined domains (e.g. a particular genre such as written academic language or blogs)
• Mostly L1 use – but can be restricted to expert use (not all L1 users have the same level of language mastery, Hulstijn, 2015; Gablasova et al, 2017)
Example: The Pearson International Corpus of Academic English (PICAE)
• 37.1M words from four science areas
• used to develop the Pearson Test in English – Academic (PTE Academic)
Corpora: Target language use (TLU)
Language testing applications
General corpora:
Particularly useful in establishing overall distribution of linguistic features, providing info about frequency of (co)occurrence
Can be used to guide test writing as well as rating scale development
Specialised corpora:
Especially useful for exploring target language domains – to understand the language use in these TLD & to obtain examples for test items or input materials
They “enable test writers to base their testing tasks more closely on authentic rather than contrived language and texts” (Taylor & Barker, 2008, p. 246)
Example: the Pearson International Corpus of Academic English (PICAE) – used to develop the Pearson Test in English – Academic (PTE Academic) (Nesi, 2016)
Corpora: L2 Exam production
L2 (learner) corpora - a group of specialised corpora representing different types of L2 production
usually contain information about speaker characteristics (e.g. proficiency, L1 background)
usually contain information about the linguistic setting (task & genre)
Exam corpora – a special category of L2 (learner) corpora representing language used in an exam context
Corpora: Exam use
Example: Guangwai Lancaster Chinese Learner Corpus • Target language: Chinese
• Over 4.2M Chinese characters
• Contains both written and spoken language from the HSK examination (Chinese proficiency test)
• Different levels of proficiency, coded for errors
Example: Trinity Lancaster Corpus – built in cooperation with Trinity College London, based on the Graded Examination in Spoken English (GESE) Over 4M words and 2,000 exam transcripts Searchable according to a number of variables
-Speaker characteristics (proficiency, age, educational level, L1 background) -Exam characteristics (task, marks for each task) -Released in 2019 (free to use)
participant roles
backchannels
filled pauses
pauses
contextual clues
Corpora: Trinity Lancaster Corpus - GESE extracts
Authenticity, language testing & corpora
Authenticity
Test development
Definition of test construct
Creation of test materials
Validation
Language in the test and TLU
Test development
Understanding of the target language use (TLU) crucial for test development and for authenticity of the test – what is required for the test-taker to perform successfully in the target environment
A variety of ways to collect primary data about the TLU used in LT - corpus findings can complement these and enhance understanding of TLU considerably – moving beyond intuitions and experience with language
Corpus contribute to authenticity in LT to two major areas in test development
Definition of the test construct
Creation of test materials
Test development: Definition of the construct
Describing the test construct – an essential step in producing effective and useful tests (Bachman & Palmer, 1996)
Corpus data – detailed description of distinguishing linguistic features of the actual language use in the target domain evidence-based definition of the construct
Distinguishing linguistic features – typical/frequent
Words & word combinations (collocations)
Grammatical and lexico-grammatical features
Description of the domains
• New domains, not yet (fully) established (e.g. blog posts, online chats, twitter messages… as new(er) genres)
• Redefine description and construct based on existing domains (e.g. academic registers).
Test development: Description of new domains
Example: Language of blogs (Grieve, Biber, Friginal & Nekrasova, 2010)
• Compiled a corpus of American blogs – 2.3M words • Identified different types of blogs and their typical linguistic properties • ‘Personal diary blogs’ – very personal voice, concerned primarily with the
blogger’s own life, also addressee focused • ‘Expert blog’ – very formal and impersonal style, focus on conveying
information on a particular topic
Implications for language testing: • How do test takers write a blog? A corpus-based study of linguistic
features in performances based on the task type ‘Blog’ (Maurer, 2018)
Further Examples of corpus-based linguistic description: Searchable web genres, e.g. opinion pieces, Q&A forums (Biber, Egbert & Davies, 2015); a legal genre – the patent specification (Groom & Grieves, 2019), the language of dark web interactions (Grieve, forth.), patterns in American English (multiple studies)
Test development: Description of existing domains
Revisiting the description of existing domains PICAE – the Pearson International Corpus of Academic English
T2K-SWAL - the TOEFL 2000 Spoken and Written Academic Language – constructed to inform development of specific tests of academic English
Re-evaluation & further description of more general constructs Römer (2017) used L1 corpora to investigate formulaic language in
spoken communication & compared it to ratings scales of major exams (the TOEFL iBT, IELTS, Cambridge English: Advanced exam)
Results: formulaic units had a major role in spoken language in the corpora – they were not consistently included in the construct definition and rating scales of these exams
Test development: Test writing
L1/EXPERT CORPORA: benefits from general & specialised L1/expert corpora for test writing - most typical (frequent and recurring) features in the TLU – to be considered for inclusion in the test development
Direct use of L1/expert corpora – source of authentic examples in the TLU – used for input materials and test items – enhancing correspondence between the language in the test and the target environment
Indirect use of L1/expert corpora – to check how similar the test items are to the authentic language use
Test validation
Eliciting language similar to TLU necessary for
Authenticity and validity of the test
Ability to predict whether the test-takers will be able to successfully communicate in the TLU
Corpora are effective at establishing the correspondence btw language use elicited in tests and the TLU
Example: Comparison of language produced in the TOEFL iBT and a corpus representing the target domain (e.g. Staples, Biber & Reppen, 2018)
Compared lexico-grammatical characteristics of texts produced by L2 users in TOEFL iBT & academic texts produced by the same writers as part of their degree (e.g. argumentative essays, lab reports)
Reported a number of differences related to linguistic features such as stance-taking (e.g. use of personal pronouns and specific verbs), hedging
Authenticity: Impact on teaching materials
The final message: Who can use corpora?
ANYONE !
What about RESOURCES & AVAILABILITY?
The final message: Who can use corpora?
1. Corpora
• Freely available corpora through Sketch Engine, CQPweb, #LancsBox
Examples: Guangwai-Lancaster corpus, Trinity Lancaster Corpus, BNC,
BNC2014, Open Cambridge Learner Corpus, EFCAMDAT, British Academic Written English (BAWE), British Academic Spoken Corpus (BASE)
• DIY Corpora: Possible to build own ones (especially from materials in electronic format) & analyse them with #LancsBox
The final message: Who can use corpora?
2. Corpus analysis, skills & training
Interfaces: more user-friendly & Software for own analysis: Freely-downloadable
#LancsBox (http://corpora.lancs.ac.uk/lancsbox/):
You can load your own corpus (in any format, it will deal with it)
Using corpus tools is easy: Wordlists, Frequencies, Collocations, Keywords, Concordancing, etc.
Who can use corpora
lancaster.ac.uk/corpussummerschools/
Who can use corpora
Starting again: September 2019
Thank you!
Alderson , J. C. (1996) . Do corpora have a role in language assessment? In Thomas, J. A. & Short, M. H. (Eds.) Using corpora for language research (pp. 248–59). London: Longman.
Bachman, L. F., & Palmer, A. (1996). Language testing in practice: Designing and developing useful language tests. Oxford University Press.
Biber, D., Conrad, S., Reppen, R., Byrd, P., & Helt, M. (2006). The Spoken and written academic language (T2K-SWAL) corpus. University language: A corpus-based study of spoken and written registers, Amsterdam: John Benjamins.
Brown, A. (2003). Interviewer variation and the co-construction of speaking proficiency. Language Testing, 20(1), 1-25.
Brown, A. (2013). Interlocutor and rater training. In Fulcher, G. & Davidson, F. (Eds), The Routledge Handbook of Language Testing (pp. 427-439). New York: Routledge.
Cushing, S. T. (2017). Corpus linguistics in language testing research. Language Testing, 34(4), 441-449.
Gablasova, D., Brezina, V. & McEnery, T. (forth). The Trinity Lancaster Corpus: Development, description and application. International Journal of Learner Corpus Research.
Gablasova, D., Brezina, V., McEnery, T. & Boyd, E. (2017). Epistemic stance in spoken L2 English: The ef fect of task type and speaker style. Applied Linguistics, 38(5), 613-637
Hulstijn, J. H. (2015). Language proficiency in native and non-native speakers: Theory and research. Amsterdam: John Benjamins.
LaFlair, G. T., & Staples, S. (2017). Using corpus linguistics to examine the extrapolation inference in the validity argument for a high-stakes speaking assessment. Language Testing, 34(4), 451-475.
Love, R., Dembry, C., Hardie, A., Brezina, V., & McEnery, T. (2017). The Spoken BNC2014. International Journal of Corpus Linguistics, 22(3), 319-344.
McEnery, A. M., & Wilson, A. (2001). Corpus linguistics: an introduction. Edinburgh University Press.
Nesi, H. (2016). Corpus studies in EAP. In The Routledge Handbook of English for Academic Purposes (pp. 230-241). New York: Routledge.
Paquot, M. (2018). Phraseological competence: A missing component in university entrance language tests? Insights from a study of EFL learners’ use of statistical collocations. Language Assessment Quarterly, 15(1), 29-43.
Park, K. (2014). Corpora and language assessment: The state of the art. Language Assessment Quarterly, 11(1), 27-44.
Römer, U. (2017). Language assessment and the inseparability of lexis and grammar: Focus on the construct of speaking. Language Testing, 34(4), 477-492.
Staples, S., Biber, D., & Reppen, R. (2018). Using corpus‐based register analysis to explore the authenticity of high‐stakes language exams: A register comparison of TOEFL iBT and disciplinary writing tasks. The Modern Language Journal, 102(2), 310-332.
Staples, S., LaFlair, G. T., & Egbert, J. (2017). Comparing language use in oral proficiency interviews to target domains: Conversational, academic, and professional discourse. The Modern Language Journal, 101(1), 194-213.
Taylor, L., & Barker, F. (2008). Using corpora for language assessment. In E. Shohamy & N. H. Hornberger (Eds.), Encyclopedia of language and education: Language Testing and Assessment (pp. 241–254). New York: Springer.
Thewissen, J. (2013). Capturing L2 accuracy developmental patterns: Insights from an error‐tagged EFL learner corpus. The Modern Language Journal, 97(S1), 77-101.
Wisniewski, K. (2017). Empirical learner language and the levels of the Common European Framework of Reference. Language Learning, 67(S1), 232-253.