Anglophone Writing Assessments: Contexts, Purposes, and Approaches

Dipping my feet into the waters

Les PerelmanMassachusetts Institute of Technology

50th Anniversary Dartmouth Institute and ConferenceFriday, August 12, 2016

Session E

Handouts at lesperelman.com

Limit my discussion today

• Advanced Placement Language & Composition Exam (USA) 2016

• AS (Advanced Subordinate) English Language (UK)

• Australian Capital Territory Scaling Test (AST) Writing

• National Assessment Program – Literacy and Numeracy (NAPLAN) – Australia

• National Writing Project. Analytic Writing Continuum (USA)

Focus on

• The purpose of the assessment

• The nature of the readings, if any

• The cognitive tasks students are asked to perform

• The time they are given to read and write

• The grading rubrics and schemes

AP® English Language & Composition Essays

• Used for college credit and college admission• Total time 135 minutes• Three essays

– Argument based on readings & data table• Readings approximately 3500 words + dense data table

– ~ 14 double-spaced typed pages– 15 minutes suggested reading time

– Rhetorical analysis of short piece (usually a speech)– Argument responding to a short quotation

• Fairly general and largely generic rubric for single holistic score

• Graded by one high school or college instructor (with back reading by table leader)

AP Question 1

Suggested reading and writing time—55 minutes.

It is suggested that you spend 15 minutes reading the question, analyzing and evaluating the

sources, and 40 minutes writing your response.

Note: You may begin writing your response before the reading period is over.

(This question counts for one-third of the total essay section score.)

AP Question 1 Continued

• Over the past several decades, the English language has become increasingly globalized, and it is now seen by many as the dominant language in international finance, science, and politics. Concurrent with the worldwide spread of English is the decline of foreign language learning in English-speaking countries, where monolingualism—the use of a single language—remains the norm.

• Carefully read the following six sources, including the introductory information for each source. Then synthesize information from at least three of the sources and incorporate it into a coherent, well-developed essay that argues a clear position on whether monolingual English speakers are at a disadvantage today.

• Your argument should be the focus of your essay. Use the sources to develop your argument and explain the reasoning for it. Avoid merely summarizing the sources. Indicate clearly which sources you are drawing from, whether through direct quotation, paraphrase, or summary. You may cite the sources as Source A, Source B, etc., or by using the descriptions in parentheses.

AP Question 2Suggested time – 40 minutes

This question counts for one-third of the total essay section score

• On June 11, 2004, Margaret Thatcher, the former prime minister of Great Britain, delivered the following eulogy to the American people in honor of former United States president Ronald Reagan, with whom she had worked closely. Read the eulogy carefully. Then, in a well-developed essay, analyze the rhetorical strategies that Thatcher uses to convey her message.

AP Question 3Suggested time – 40 minutes

This question counts for one-third of the total essay section score

• In 1891, Irish author Oscar Wilde (1854–1900) observed, “Disobedience, in the eyes of anyone who has read history, is man’s original virtue. It is through disobedience that progress has been made, through disobedience and through rebellion.”

• Wilde claims that disobedience is a valuable human trait and that it promotes social progress. Write an essay that argues your position on the extent to which Wilde’s claims are valid. Use appropriate examples from your reading, experience, or observations to support your argument.

UK AS (Advanced Subordinate) English Language

• Used for university admission along with later A Level Exam• Two “papers”

– 90 minutes each– 2-3 essays per paper– Essays based on short readings & data tables in various media

• Explicit Assessment Objectives• Comprehensive grading multi-trait rubrics explicitly linked

to assessment objectives– Some assessment objectives conflate several traits together into

what could be considered a semi-holistic score

• Graded by single secondary or tertiary instructor

UK English AS LanguagePaper 1: Language and the Individual

(90 min.)

UK English AS LanguagePaper 1: Language and the Individual

Marking Scheme

UK English AS LanguagePaper 2: Language Varieties

(90 min.)

Question 3

UK English AS LanguagePaper 2: Language Varieties

Marking Scheme

USA AP Texts, Questions & Rubrics vs. UK AS Texts, Questions, & Rubrics

• USA AP Texts

– Requires rhetorical knowledge

– High culture

– Politically safe – Margaret Thatcher eulogy of Reagan

– Wilde now part of Canon

• UK AS Texts

– Requires extensive socio-linguistic knowledge

– Language students encounter

– Much broader social and political landscape

Australian Capital Territory Scaling Test (AST) Writing

Developed by Australian Council on Education Research

• Used for tertiary admission– Merged with other State and Territory Tests to

calculate a single Australian Tertiary Admission Rank

• Questionless question– No direction on formulating a thesis

• Given considerable and diverse data and then asked to argue about the issue raised in the material

• Each paper is graded by four readers (all instructors)– Using 1-10 holistic scale

AST W Writing Prompt

ACER Criteria for Assessment of Written Expression

• THOUGHT AND CONTENT (the quality of what is said in the piece of writing)– what is made of and developed from the task– the kinds of thoughts and feelings offered in response to the task

• STRUCTURE AND ORGANISATION (the quality of the structure and organisation developed to present what is said in the writing)– the shape and form of the piece– the sequence and cohesion of the piece

• EXPRESSION, STYLE AND MECHANICS(the quality of the language used to organise and present what is said)– the effectiveness and appropriateness of the language– the expressiveness and fluency of the language– the control of the mechanics of English

National Assessment Program –Literacy and Numeracy (NAPLAN)

• Given to 3, 5, 7, & 9 years

• Two prompts (either narrative or persuasive):– 1 for 3 & 5;

– 1 for 7 & 9

• Students have 40 minutes to plan, write, and revise

• All essays are graded by two teachers using the same 10 trait rubric

NAPLAN Purpose

• Formative check on student ability

• Assessment mechanism for

– Teachers

– Schools

NAPLAN Persuasive Essay Prompt

NAPLAN Trait Categories

NAPLAN Persuasive Marking GuideTop Score Examples (In Handout)

• Audience– “things should be regulated”

• Text Structure– “things should be regulated”

• Ideas– “things should be regulated”

• Persuasive devices– “things should be regulated”

• Vocabulary– “things should be regulated”

NAPLAN Persuasive Marking GuideTop Score Examples -- continued

• Cohesion– “things should be regulated”

• Paragraphing– “things should be regulated”

• Sentence Structure– “things should be regulated”

• Punctuation– “things should be regulated”

• Spelling– “things should be regulated”

Encourages formulaic five paragraph essays

• One of the better concluding paragraphs:– There are both positive and negative effects on animals,

concerning keeping them in cages, making this a highly debated topic amongst the general public. I believe the best option is to continue keeping them in cages but making sure the cage is a good size in relation to the animal and try to replicate the animals natural habitat and behavior as much as possible. It is important to preserve endangered species both for the sake of the animal and environment, and for future generations. But we have to make sure we do this in the best possible way, a way in which both the animals and general public are happy.

– http://www.vcaa.vic.edu.au/Documents/naplan/writingtest/STUDENT_10.pdf

NAPLAN to Use AES To Score All 10 TraitsOn Tuesday Dr Rabinowitz from the Australian Curriculum, Assessment and Reporting Authority (ACARA) revealed that computers would mark the writing of a million Australian students in year 3,5,7 and 9 from 2017 onwards.

Under the plan the computers would be fed a thousand sample essays marked by teachers to create an algorithm that could then mark the rest of the students in the state.

Dr Rabinowitz said the artificial intelligence system would identify "sentence structure, vocabulary, grammar, right down to more mundane elements like spelling" and can perform "as well as or even better than the teachers involved".

– Sydney Morning Herald April 30, 2015

Who is Stanley Rabinowitz?

• Formerly, Senior Program Director of WestEd’s Assessment & Standards Development Services (ASDS), a program of national scope with more than 100 staff throughout the USA. In that role, he served as Director of the national Center for Standards and Assessments Implementation (CSAI) and the Smarter Balanced Assessment Consortium Project Management Partner (PMP).

• Dr Rabinowitz was also a member of the Common Core State Standards national validation committee. Prior to joining WestEd, Dr Rabinowitz served as State Assessment Director for the New Jersey Department of Education.

National Writing Project

• Analytic Writing Continuum– Purpose: Curriculum & program evaluation– Holistic score, plus– Analytic scores

• Content• Structure • Stance• Sentence Fluency• Diction• Conventions

– Expensive• Expert graders needed• 1½ days of training

– Accurate– 7 dimensions of information– Designed for 3-12, but has been successfully adapted for major college

assessments

Summary

• most other English speaking countries often assess subjects by asking to students to write out long answers, making writing an integral part of learning throughout the curriculum

• some prompts in other English speaking writing assessments are open-ended

• most other English speaking assessments give students at least 90 minutes and often more to respond to one or more essay questions. not 20 minutes

• some English speaking countries give word or page guidelines for essays to minimize the effect of word length on scores and to promote concision

• most other English speaking countries have essay assessments graded by teachers as a recognized and compensated form of professional development

However

• The worst characteristics of American writing assessment are sometimes but not always infecting our Anglophone cousins – in a sort of Gresham's law – of bad cheap writing assessment driving out good expensive writing assessment

• The Writing Assessment equivalent of

The Preventive: Good Research

Final Report Major Findings• SAT® scores and total A level (or equivalent) score points were generally related

in a similar way to a number of educational and socio-economic factors, including the type of institution attended, ethnicity, eligibility for free school meals (FSM), etc.

• Female students had higher total GCSE and A level points scores and achieved significantly higher scores on the SAT® Writing component than male students. Male students performed significantly better on the SAT® Mathematics component and on the SAT® as a whole.

• Students who achieved very high SAT® scores were not always the same students who achieved three or more A grades at A level and vice versa. Students who achieved very high SAT® scores but did not form part of the ‘three A grades’ group tended to be male students.

• Two regression analyses, one controlling for A level total scores and one controlling for both A level scores and average prior attainment at GCSE, showed that female students, some ethnic minorities, students with special educational needs (SEN) and students learning English as an additional language (EAL) appeared to perform less well on the SAT® than expected compared to default categories (i.e. boys, white students, etc).

Or

• The American SAT does not improve on UK assessments

• And

• It underreports scores for women, certain ethnic groups, and English language learners.