Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
2015/6/3
1
复旦大学
复旦大学Assessing English for Academic Purposes (EAP):
Construct Definition, Current Practice, and Future Development
Presenter: Jinsong FAN, Ph.D. Fudan University
Email: [email protected], May, 2015
复旦大学
Outline of Presentation
1. Language Assessment as “the Art of the Possible”
2. Understanding the Construct: What is EAP?
3. The Application of Socio-Cognitive Validity Framework in EAP Assessment: The Case of Academic Listening
4. Assessing EAP in the Classroom: English Translation Course (an EGAP Course)
5. The Future of EAP Assessment
2
复旦大学
Language Assessment: The Art of the Possible
Deciding the purpose of assessment
Understanding the construct
Operationalizing the construct
Giving feedback
3
Placement Achievement Proficiency Diagnostic
Theory in linguistics & AL
Pedagogical experience
MCQ SAQ Essay writing …
Numeric score Grade Description …
复旦大学
Fundamental Considerations in Language Assessment
4
Reliability
Construct validity
Authenticity
Interactiveness
Impact
Practicality
TEST USEFULNESS
Test Usefulness Framework, from Bachman & Palmer, 1996
复旦大学
Assessing EAP: From Large-Scale Testing to Classroom Assessment
Classroom-based EAP assessment: Essay writing, presentation, portfolio, quiz…
Localized EAP assessments: TEPT, FET, TOPE…
Large-scale national EAP assessments: CET-6, TEM-4 & 8, PETS-4 & 5…
Large-scale international EAP assessments: IELTS, TOEFL, and PTE…
5 复旦大学
Understanding the Construct: What is EAP?
EAP has emerged out of the broader field of ESP, a theoretically and eclectic parent, but one committed to tailoring instruction to specific rather than general purposes (Hyland & Hamp-Lyons, 2002, p. 2).
EAP is defined as teaching English with the aim of facilitating learners’ study or research in that language (Flowerdew & Peacock, 2001; Jordan, 1997).
6
2015/6/3
2
复旦大学
ESP
EOP EAP
Airlines Medicine EGAP ESAP
PilotsAir traffic
controllers
EST Liberal Arts
Science Technology Humanities
Doctors Nurses
Social Sciences
The ESP hierarchy with examples of courses (from Clapham, 2000, p. 513)7 复旦大学
Academic English is a variety or register of English used in professional books and characterized by the specific linguistic features associated with academic disciplines… Academic English tasks include reading abstracts, getting down the key ideas from lectures, and writing critiques, summaries, annotated bibliographies, reports, case studies, research projects, expository essays. It includes a wide range of genres (Scarcella, 2003, p. 9).
8
复旦大学
EAP Vs. EGP: Linguistic and Cognitive Features
PROFICIENCY PROFICIENCY
Higher Order Thinking
Higher Order Thinking
Metalinguistic Abilities
Metalinguistic Abilities
Phonological Features
Phonological Features
Grammatical Features
Grammatical Features
Vocabulary Features
Vocabulary Features
Sociolinguistic Features
Sociolinguistic Features
Strategies Strategies
Background Knowledge
Discourse FeaturesDiscourse Features
Background Knowledge
Writing an expository essay Vs. Participating in everyday conversation,
from Scarcella, 2003, p. 26-27 9 复旦大学
The Case of Academic Listening
1. Taxonomies of listening skills, sub-skills, and strategies (e.g., Buck, 2001; Richards, 1983; Weir, 1993)
2. Conduct needs analysis/assessment to understand the TLU domain (e.g., Weir, 1983, 1990)
3. Basic and higher level skills in communication and literacy: The distinction between basic interpersonal communication skills (BICS) and cognitive academic language proficiency (CALP) (Cummins, 1980)
4. Can do statements / descriptors (e.g., Common European Framework of Reference – Council of Europe, 2001)
10
复旦大学
Taxonomies of Listening Skills
11
Conversational ListeningEGP
LISTENING ABILITY & SKILLS
Academic ListeningESP - EAP
Ability to discriminate between the distinctive sounds of the target language
Ability to recognize stress patterns of words
Ability to process speech at different rates
…
Ability to identify purpose and scope of lecture
Ability to identify topic of lecture and follow topic development
Familiarity with different registers
Knowledge of classroom conventions
…
复旦大学
Taxonomies of Academic Listening
1. Ability to identify purpose and scope of lecture
2. Ability to identify topic of lecture and follow topic development
3. Ability to identify relationship among units within discourse (e.g., major ideas, generalizations, hypotheses, supporting ideas, examples)
4. Ability to identify role of discourse markers in signaling structure of a lecture (e.g., conjunctions, adverbs, gambits, routines)
5. Ability to infer relationships (e.g., cause, effect, conclusion)
6. Ability to recognize key lexical items related to subject/topic
7. Ability to deduce meanings of words from context
8. Ability to recognize markers of cohesion
9. Ability to recognize function of intonation to signal information structure (e.g., pitch, volume, pace, key)
10. Ability to detect attitude of speaker toward subject matter
11. Ability to follow different modes of lecturing: spoken, audio, audio-visual 12
2015/6/3
3
复旦大学
12. Ability to follow lecture despite differences in accent and speed
13. Familiarity with different styles of lecturing: formal, conversational, read, unplanned
14. Familiarity with different registers: written versus colloquial
15. Ability to recognize relevant matter: jokes, digressions, meanderings
16. Ability to recognize function of non-verbal cues as markers of emphasis and attitude
17. Knowledge of classroom conventions (e.g., turn-taking, classification requests)
18. Ability to recognize instructional/learner tasks (e.g., warnings, suggestions, recommendations, advice, instructions)
13
Source: Richards, 1983, see also Buck, 2001
复旦大学
1. Ability to identify purpose and scope of lecture
2. Ability to identify topic of lecture and follow topic development
3. Ability to identify relationship among units within discourse (e.g., major ideas, generalizations, hypotheses, supporting ideas, examples)
4. Ability to identify role of discourse markers in signaling structure of a lecture (e.g., conjunctions, adverbs, gambits, routines)
5. Ability to infer relationships (e.g., cause, effect, conclusion)
14
复旦大学
12.Ability to follow lecture despite differences in accent and speed
13.Familiarity with different styles of lecturing: formal, conversational, read, unplanned
14.Familiarity with different registers: written versus colloquial
15 复旦大学
Examples of CEFR Descriptors
16
复旦大学17 复旦大学
TEST TAKER CHARACTERISTICS Physical / Physiological Psychological Experiential
TASK SETTING Response method Weighting Knowledge of criteria Order of items Channel of presentation Text length Time constraints, incl.
number of listenings
SETTINGS: ADMIN Physical conditions Uniformity of admin Security
LINGUISTIC DEMANDS: TASK INPUT AND OUTPUT
Overall text purpose Speaker-listener
relationship Discourse mode Functional resources Grammatical resources Lexical resources Nature of information Content knowledge
INTERLOCUTOR Speech rate Variety of accent Number + gender
CONTEXT VALIDITY
COGNITIVE PROCESSES Goal setting Decoding acoustic/visual
input Syntactic parsing Establish propositional
meaning Inferencing Building a mental model Creating a text-level
representation Monitoring comprehension
COGNITIVE VALIDITY
RESPONSE
SCORE/GRADE
SCORING VALIDITY Item difficulty Item discrimination Internal consistency Error of measurement Marker reliability Grading and awarding
Using the Socio-Cognitive Model (Weir, 2005)
for Academic Listening Assessment
Development & Validation
18
2015/6/3
4
复旦大学
In assessing academic listening, “at least some of test tasks should require test takers to engage not just lower level comprehension processes (lexical access, syntactic parsing, propositional meaning at clause/sentence level), but also high-level processes that force them to integrate and reconcile incoming information, possibly from different sources, with their own knowledge sources (topical, genre, rhetorical, etc.) in order to create a text-level or inter-textual presentation or mental model (Taylor & Geranpayeh, 2011, p. 97).
19 复旦大学
EAP Classroom Assessment in Practice: English Translation Course
1. English Translation (ET) Course as an EGAP course
2. ET course delivery: Teaching content and method
3. Learner characteristics
4. ET course assessment
20
Assessment Context
Assessment Method
复旦大学21
The position of ET course in the teaching system
复旦大学
The overarching goal of this courseTo further improve students’ EGAP ability and skills, especially in terms of reading and writing, through the study of translation
1. Develop some basic understanding about the nature, process, and product of translation
2. Learn some basic skills and strategies in conducting C-E and E-C translation
3. Learn how to critique a translation based on what they have learned in class
4. Learn how to conduct a translation mini-project and present the relevant research findings
Course objectives at the general and specific levels
复旦大学
1. Lecture (Translation theory & practice)
2. Review of students’ translation assignments (Translation skills & strategies)
3. Translation critique (Critical thinking, seminar discussion)
4. Translation mini-projects (Academic writing, oral presentation)
The four components in ET course delivery
复旦大学
Overall Design of ET Course Assessment
Component Method Participant Target Construct
FormativeAssessment (FA)
Translation assignment
Teacher Translation skills
Translation project (written)
TeacherStudent
Academic writing
Translation project (oral)
TeacherStudent
Oral presentation
Translation critique (written)
TeacherStudent
Academic writing Critical thinking
Translation critique (oral)
TeacherStudent
Oral presentation
Classroom quiz Teacher Translation skills
SummativeAssessment (SA)
Final examination Teacher Academic writing Critical thinking Translation skills
24
2015/6/3
5
复旦大学
Course evaluation
Translation critique 10%Students read a translation work out of class, and write a short reading report. Students need to discuss their reading experience in the seminar session.
Translation mini-projects
20%
Students work in groups on a translation mini-project(mainly out of class), and then present the relevant findings, both orally and in the written form, in class. Peer review constitutes part of the evaluation.
Translation assignments
20%
Each student is required to complete the translation assignments of 8 texts, 4 from English into Chinese and the other 4 from Chinese into English after class. Peer review constitutes part of the evaluation.
Final examination 50%
Students are required to comment on a given translation, and complete the translation of two texts, one from English into Chinese and the other from Chinese into English.
复旦大学
Examples of translation assignment
E-C translation
Example 1: The Difficulty of Translation
Example 2: How I Escaped the Reform School
C-E translation
Example 1: Language and Social Identity
Example 2: Flowers and Air-Conditioners
复旦大学
Translation Mini-Projects (FA)
Students are required to work in groups (3-5 per group) on two translation mini-projects based on the project description;
After working for two weeks on the project, students are required to submit a research report (written in English around 1,500 words) to describe how the project is undertaken and to summarize the major findings;
Students need to present orally how they have conducted the project and the major research findings.
复旦大学
Examples of students’ written reports
Example 1: Poetry translation (from the 2013 spring cohort)
Marking criteria
Example 2: The role of purpose in translation (from the 2014 spring cohort, TT & Report)
Example 3: Yan Fu’s translation (from the 2013 autumn cohort)
Teacher’s feedback
复旦大学
An example of reading assignment (C-E)
Chapter 1 of Fortress Besieged, written by Ch’ien Chung-shu and translated by Kelly & Mao 复旦大学
An example of reading assignment (E-C)
Animal Farm, written by George Orwell and translated by Fu Weichi
2015/6/3
6
复旦大学
Is it possible to adopt a combination of teacher-, self-, and peer-assessment in ETC: Students’ views
Items Min Max Mean SD
1. I can assess my own translation work responsibly. 3.00 6.00 5.16 0.79
2. I can assess my others’ work responsibly. 4.00 6.00 5.35 0.63
3. I think it necessary for me to assess my own work. 1.00 6.00 4.37 1.29
4. I think it necessary for me to assess others’ work. 1.00 6.00 4.95 1.14
5. I can better understand the rating criteria better through assessing my own translation work. 1.00 6.00 4.61 1.17
6. I can better understand the weaknesses of my translation through assessing my own translation work. 1.00 6.00 4.88 1.08
7. I can better understand the rating criteria better through assessing others’ translation work. 3.00 6.00 5.36 0.70
8. I can better understand the weaknesses of my translation through assessing others’ translation work. 3.00 6.00 5.53 0.65
复旦大学
Peer-assessment form for oral presentation
32
复旦大学
Rationales for further study
1. Doubts over the reliability and validity of self- and peer-assessment;
2. Most rating scales are designed for standardized translation tests, but very few scales are available for teacher-, self-, and peer-assessment in translation teaching;
3. Previous studies are either qualitative or adopting the true score theory. Many-Facet Rasch Analysis (MFRA) can address the limitations of statistical analyses in previous studies.
复旦大学
Two Recent Projects
1. The development and validation of analytic rating scale for translation teaching (2013ZD05, Fudan University Key Research Project)
2. Teacher-, self-, and peer-assessment in translation teaching: A many-facet Rasch modeling approach (2014SH008, National Higher Education English Teaching Research Project)
34
复旦大学
A new rating scale?
An example of the rating scale
Reliability
Construct validity
Practicality
Diagnostic feedback
Teachers’ and students’ views
复旦大学
The Future of EAP Assessment
1. A clearer understanding of the EAP construct being assessed: Construct underrepresentation (CU) and construct-irrelevant variance (CIV) as two major threats to test validity (see Messick, 1989, 1996)
2. The increasing role of technology in assessing EAP: Assessing EAP listening (input mode); assessing EAP writing and speaking (task delivery, automated scoring, e.g., Xi, et al., 2008; Xi, 2010); assessing integrated EAP ability and skills (e.g., Yu, 2013)
36
2015/6/3
7
复旦大学
Connecting EAP Teaching, Learning, and Assessment
1. Developing localized EAP assessments, and use them in combination with large-scale national/international EAP assessments (e.g., Fan, Ji, & Song, 2014; Fan & Song, 2015): Content & construct validity, relevance, and washback
2. The development of EAP/ESP ability descriptors which tailor to the needs of EAP/ESP teaching and learning in China (e.g., Jin, Zhang, & Zou, 2015)
3. The development of diagnostic EAP assessments, using the state-of-the-art Cognitive Diagnostic Assessment (CDA) models in language assessment (e.g., Read, 2008; Jang, 2009; Lee & Sawaki, 2009)
37 复旦大学
References
38
复旦大学
复旦大学
39复旦大学
复旦大学
Jinsong (Jason) FAN, Ph.D
Language Testing Centre, Fudan University