The New SHAPE Listening & Reading Tests

OR

HOW DID THEY DO THAT?!

BILC Seminar – Stockholm, SwedenOctober 2013

Michael Adubato SHAPE Language Testing Centre

Belgium

GERMANY 5 – 3 SWEDENFriends Arena, Stockholm

16 October 2013

Designing tests for SHAPE – a true learning process

University of Westminster

Aim

To explain the challenges we faced designing

the Listening and Reading tests and how we

overcame them.

The Format – Listening and Reading

SLP 2: 20 x passages20 x MCQ

SLP 3: ? x passages20 x MCQ

SLP 4: ? x passages 20 x MCQ/ SA

SLP 2 – 3Listening: 60 minsReading: 90 mins

SLP 3 – 4Listening: 75 minsReading: 120 mins

Listening:Speed per levelRubricOrientation messageTime to complete MCQ

ReadingReading speed per levelTypes of reading: skim/ scam/ read for detail

Factors affecting Passage Length

Required test length

Required test length

Validity

Contexts English Genres

Paper question

Task Focus Category Paper A Topic Word Length

English Answer Author

1 Answer who, what, when questions

Pers Serial 5: Invitation to party Scottish Dancing 109  British a KW

2 Answer how and why questions

Work Serial 31: Travel order Brussels Airport 127  USA b RW

3 Answer how and why questions

Gen Newpaper article Sleep patterns 119 British  a RW

4 Answer how and why questions

Work Equipment request forms New system for booking equipment

124  USA a MR

5 Identify explicity stated main ideas

Work Serial 4: Routine office email exchange

Revised HR policy requiring checking for legality

112  Canadian c RW

6 Distinguish fact from opinion

Work Serial 4: Routine office email exchange

Brown and Laing Account 106 Australian  b CH

7 Answer who, what, when questions and how,why questions

Work Serial 40: Request for translator/interpreter services. Include details of requirement

Interpreter required for Ukrainians 166  USA c RW

8 Interpret complex sentences

Work Serial 11: How to use the airconditioning

Fire safety 210  British c JAG

Sample from Examination Schematic

Creating the texts………

Which factors affect the complexity of a text?

• Task focus

• Criteria eg STANAG or CEFR

• Grammar• Lexis • Sentence length• Density of text/passage

• Speed (listening)• Interference (listening)• Level of inference

Answer who, what, when questions

Answer how and why questions

Distinguish fact from rumour or hearsay

Recognise author's intentions

Recognise attitudes and emotions of the writer

Outline the development of an argument

Task Foci

Writing and Setting Process

Undertake TNA

Allocate test items to writers

Train writers Scrutinise in-office

Writer edits

Internal Setting Board

In-house edits

Pilot papers

Produce paper

Produce Schematic

Allocate items to writers

Scrutinise in-office

Writer edits

Order tasks

Initial draft

Internal Setting

Pilot and adjust

Select writers

Select final writers

Recording

Why do people offer to sit a pilot examination?

For a reward

Joy

To please others

They have no reason to do so

Piloting Exam Paper in the UK

Piloting the paper overseas

SLP 2/3/4??

What did SPSS do for us?

(Remember, items increase in difficulty throughout each level of the test.)

Enabled us to see how many people answered a question with a, b, c or d.

Pre-test questionnaires

– Gender

– Nationality

– Language level (variation required)

– Educational level

NOT SHOWN!IF YOU’RE INTERESTED IN THIS, I CAN PROVIDE YOU

WITH A COPY…

Demonstration of Paper C Spreadsheet

Questions?