Embed Size (px)
Citation preview
Conflicting demands in Conflicting demands in integrated reading / writing integrated reading / writing taskstasks
Tom Lumley Australian Council for Educational Research
Annie Brown University of Melbourne
2
Typical Tests of English for Typical Tests of English for Academic PurposesAcademic Purposes
eg• TOEFL - Test of Written English (USA)• IELTS (UK/Australia)
Writing tested as discrete skill
Essay format
Decontextualized, short prompts
No specific content
Students’ background knowledge / opinion
3
New TOEFLNew TOEFL
• Prototype LanguEdge Writing tasks
Integrated Reading / Writing
Reading Input text (c. 750 words)
followed by related Writing Task
plus
Independent (stand-alone) Writing task
(LanguEdgeTM Courseware Handbook for Scoring Speaking and Writing.
© ETS, 2002.)
4
Task instructionsTask instructions
• You have 25 minutes to answer the question below by writing a response based on information from the passage. Typically an effective response will be between 175 and 250 words.
• Your response should present the key ideas and supporting points from the passage necessary to answer the question fully in your own words as much as possible.
• Your response will be scored on the quality of your writing and how well and how completely you convey the relevant information from the passage. Typically an effective response will be between 175 and 250 words.
(LanguEdge Courseware Handbook for Scoring Speaking and Writing. ETS, 2002.)
5
TasksTasks
Task 1: “Early Cinema”Explain how projectors changed the economics of showing films and the experience of watching films.
Task 2: “Nineteenth-century politics in the United States”
Explain the different viewpoints held by nineteenth-century Whigs and Democrats on the economy and the role of government and explain why different people supported each of the two parties.
(LanguEdge Courseware Handbook for Scoring Speaking and Writing. ETS, 2002.)
6
Assessment criteriaAssessment criteria
holistic scale:
• relevance, accuracy• clarity and completeness of ideas• organisation and linking of ideas
• linguistic accuracy• paraphrasing and appropriate reference to
input text
7
Test TakersTest Takers
• written texts - 60 learners of English
n Language Institution place
30 Mandarin University Beijing
20 Korean Lang. centres Melbourne
10 Cantonese University / univ. prep.
Hong Kong/
Melbourne
8
Verbal report dataVerbal report data
• Test takers:Interviews immediately following writing
Collected in L1, translated into English
• 6 ratersVerbal reports as they assessed the texts
produced by the test takers
9
Test taker interpretations of the task Test taker interpretations of the task requirementsrequirements
What does the task require ?
•Do test takers consider them to be principally reading or
writing tasks?
•Do they recognize that they are not required to summarise
the whole text?
•How do they think their responses are assessed?
•Are they more concerned about quality of the content or of
the language they produce?
•How do their views affect the way they approach the task?
10
Findings: task requirementFindings: task requirement
MC09
I should only be summarising and paraphrasing. … I was quite
selective, I tried to look for the relevant parts in the text which
supported what I wanted to write.
BE05It expects me mainly to summarize the central ideas of the
original text and then to re-write it for short. A re-writing of the main content.
• Rater 6 BE05I think this student has done a similar thing to the previous one
in that they don’t seem to be addressing the question. On first go a lot of it’s irrelevant.
11
Findings: task requirementFindings: task requirement
• MC08I planned the structure of my essay in my head . I did not think it
was necessary to draft as the question is quite clear, and I knew what to write.
• Rater 3 MC08principal ideas are inconsistently presented. There’s some
irrelevant stuff in there and it’s not explicit enough about the
key terms of the question.
12
Findings: Findings: text structuretext structure
• MK23My text is a typical organization of an essay, which includes introduction, body and conclusion.
• MC11(in the introduction) I paraphrased the question. … I think it is better to include an introduction. … The purpose of the introduction is … to give the reader some idea of what I am about to write.
• B14
Q: Did you have the introduction part?A: No. I didn’t even think about it.
13
Findings: Findings: text structuretext structure
• HK01Q:Ok, so how did you go about organizing this text?A: No need to organize because there are 3 parts in the
question, so I just wrote one paragraph for each part.
• HK07How can we complete the answer in 250 words only?
• BE07 My conclusion part was simply a repeat of the introduction, not
a conclusion based on the different paragraphs in my writing.
14
Findings: Findings: text structuretext structure
• Rater 5 BE04I would like an introduction. I don’t really know what they’re talking about, … so I am a bit confused when I start reading it. …. if it had a context it would receive a higher mark from me
• Rater 2 MC06This one I’m afraid really has to be a 1 because it spends almost a third of its time saying what it’s going to do and then has very little …
• Rater 3 B10
If I look at organisation. It’s been organised so that the first
sentence … rephrases the question, which is quite a
reasonable way to go about it.
15
Assessment criteriaAssessment criteria
• Your response will be scored on the quality of your writing
and how well and how completely you convey the relevant
information from the passage. Typically an effective response
will be between 175 and 250 words.
(LanguEdge Courseware Handbook for Scoring Speaking and Writing. ETS, 2002.)
16
Assessment criteria: content Assessment criteria: content focusfocus
• MC11I feel that the content is more important, because the task is trying to find out if I understand the passage.
• MC13I focused on getting the ideas right, because if the ideas were incorrect, even if I wrote beautifully, it would still be useless.
• MC06Q: Which is more important to you in answering this task, do you think – content or language?
A: Getting the ideas right, I didn’t have any time to think of my language.
17
Assessment criteria: content Assessment criteria: content focusfocus
• MC08I think getting the ideas right is more important. If you are asked to answer some questions based on a text, you are expected to provide the correct answers, you can’t just write anything. Even if your English is good, it doesn’t count.
• MC06I think the marker will focus on content, he or she will use a checklist and check if I have listed all the points required to answer the question.
18
Assessment criteria: Assessment criteria: real worldreal world
• B09At my study level, accurate language is not a big issue. The key is to have correct ideas.
• MC05Because I don’t know the criteria, so I was sort of guided by the assessment criteria for an academic paper.
19
Assessment criteria: Assessment criteria: language focuslanguage focus
• B10
Q: Which one was more important to you, correct ideas or accurate language?
A: Accurate language. After all, it was a test of writing ability, that is, a test of language.
• B15
Q: Which is more important to you in answering this task, do you think – correct ideas or accurate language?
A: Language, because I feel little difficulty with getting correct ideas.
20
Task completion: copyingTask completion: copying
• BE08
I have copied some sentences, and some phrases, and also
the ideas. I don’t think this is a writing task, but more like a
paraphrasing task, or summarizing task.
• HK01Q : Are you happy about the way you have (copied)?
A : Of course not, but I had no time.
21
Reasons for copyingReasons for copying
BE01
I felt if I changed the words, the meaning of my writing would change. Therefore, I prefer to use the expressions in the original text.
B07
I felt those words fit in very well, and I could find no better words than them.
HK01
“the proper sphere of government action” … “moral beliefs” … I don’t understand those expressions, so I copied the whole sentence to my draft.
22
Difficulties of paraphrasingDifficulties of paraphrasing
• MC05
I was inclined to use my own words than direct paraphrasing. I
find that it is actually easier to use your own words than
paraphrasing, because in paraphrasing you have to find the
right words to express the writer’s ideas, I think that is even
more demanding.
23
RatersRaters: : copied textcopied text
• Rater 5 BE01 So, I don’t get a sense of this person at all because there is too much
of it lifted from the original text…
• Rater 3 B17
the first sentence “projectors largely changed the experience of
watching films” - I’m not sure whether that’s been lifted or
not, but it reads outside the writer’s own voice a bit.
24
RatersRaters: : copied text?copied text?
• Rater 3 MC09
I’m just wondering about how much is plagiarised. Things
like “Also, the economics of showing films increased
because the image could be projected to a large screen for
mass audience ” That seems like it’s come from the text
and I can’t find it. But it seems like it should have, I’m sure
I’ve read that.
25
RatersRaters: : what are we testing?what are we testing?
• Rater 5 B093
and I guess that’s the tension where we’ve got to make up our
mind, what are we testing? Are we testing understanding, or
are we testing language? And so if we’re testing understanding
and the student is able to lift the sentence and then put a
sentence in their own text that explains and elaborates the
previous sentence then can I reward the student for having a
good understanding, and being able to select appropriately from
the text and then elaborate in their own words? It’s a bit of a
tension I face with this one I reckon. So overall though, now
goodness me I have to score it, it’s a problem.
26
Raters: Raters: in praise of paraphrasein praise of paraphrase
• Rater 5 B053 the student, I think, has got an understanding of the structure
of the text … I mean she has attempted to use her own words
a little bit, I think, rather than just lift passages from the text as
I’ve seen before. So, it hasn’t been successful, but she has
tried to paraphrase it herself. So, again, big decisions. I think
there is enough understanding of the content even though
some of the sentences don’t quite make sense.
27
Advice to test takersAdvice to test takers
• a full summary is not required or appropriate
• an introduction is required
• conclusion is not required
28
The task: The task: a trade-offa trade-off
• completing the task by relying on the
language of the input text
OR
• using their own words and hence reducing the quality of the language.
29
Advice to test takers?Advice to test takers?
• focus on content:– too much of a temptation to copy– conflicts with the need to use their own words and
demonstrate their own competence in English
• focus on paraphrase and their own language:– time constraint– difficulty of fulfilling task successfully– memory load
• edit and proofread their work– time constraint
30
ConclusionConclusion
• need to redesign the assessment task
Task Input
or
Task focus & instructions
(for information on current TOEFL tasks see www.toefl.com )
