Using text-to-speech in exams - practical solutions and pitfalls, UK perspective

  • View

  • Download

Embed Size (px)


Presented by Paul Nisbet and Abi James at the BDAN International Conference, March 2014. Since 2012 the JCQ Access Arrangements have acknowledged that candidates using a Computer Reader or text-to-speech technology are reading independently making such provision available to candidates in exams that test reading skills for the first time. While use of digital exams with text-to-speech has been widely supported in Scotland through the work of SQA and CALL Scotland for a number of years, the rest of the UK has not had equivalent access. From 2013/14 exam boards in England, Wales and Northern Ireland propose to provide digital versions of exam papers to schools for text-to-speech users. This paper will draw on experiences in Scotland and the work of the BDA New Technology Committee to identify processes and best practices within schools for using these digital papers and to identify the best text-to-speech technology to maximise the benefits for students.

Text of Using text-to-speech in exams - practical solutions and pitfalls, UK perspective

  • 1. Paul Nisbet, CALL Scotland University of Edinburgh Abi James, BDA USING TEXT-TO-SPEECH IN EXAMS - PITFALLS AND PRACTICAL SOLUTIONS

2. Reading difficulties in secondary 20% of 11 year olds have poor reading comprehension (data from SAT results) by 14 years of age 33% of pupils have unsatisfactory reading comprehension Chris Singleton, Special Children 182 April/May 2008 3. While four-fifths of pupils at Key Stage 2 reached national expectations over the last three years, one in five primary pupils did not achieve the expected standard in English. Moving English forward: action to raise standards in English, Ofsted 2012 Reading difficulties in secondary 4. SQA Assessment Arrangements Assessment arrangements allow candidates who are disabled and/or who have been identified as having additional support needs appropriate arrangements to access the assessment without compromising its integrity. Introduction to Assessment Arrangements for Schools and Colleges SQA January 2010, revised June 2013 5. Assessment Arrangements 2013 Requests for Assessment Arrangements made for Scotland: 17,263 candidates (11.3% of candidates) Rest of UK: 145,430 candidates (6.9%) Requests made for 61,680 entries (exams) (8.3% of all entries in Scotland) 6. Assessment Arrangements 2013 Scotland Rest of UK Requests for Access Arrangements 17,263 candidates (11.3%) 145,430 candidates (6.9% - many arrangements delegated to schools) Extra Time 76% 59% Reader 30% 27% Scribe 24% 15% Use of ICT including digital papers 19% n/a - use of word processing delegated to schools Computer readers 2% 0.4% Coloured paper (excludes coloured overlays) 3% 2% 7. Computer Reader (TTS) vs. Human Reader TTS offers an independent means to decoding text. TTS provides greater consistency. Human readers can be more flexible (can read anything). Some learners prefer to use TTS. Studies have shown candidates are more likely to check questions with TTS than using a human reader, resulting in higher scores (Dolan et al, 2005) Computer Readers are allowed in ALL exams, including all GCSE, A-level & Functional Skills exams including those testing reading skills since it allows the candidate to independently meet the requirements of the reading standards 8. A 6th year pupil at Denny High in Falkirk was assessed using the Neale Analysis, reading with and without text-to-speech. Reading herself, her comprehension age was 6 years 9 months. With text-to-speech, it was over 13 years. 9. 2002 need for Digital Question Papers with TTS identified 2003-04 research into specification; development of papers 2004-05 evaluation in 6 schools 2005-06 pilot trials #1 - 31 students used digital papers in 105 examinations 2006-07 pilot trials #2 - 80 candidates used 490 digital papers in 200 examinations 2014 Rest of UK PDF papers will be provided by exam boards (not interactive); UKAAF guidance The Scottish Experience: Research, Development and Trial 10. Type your answers Click to tick Use on-screen drawing tools Read questions with text-to-speech SQA Digital Question Papers 11. Software and Apps Windows Acrobat Reader (free) Text-to-speech: ClaroRead, Co:Writer, Ivona MiniReader , Penfriend, Read and Write Gold, Adobe Read Out Loud, etc iPad ClaroPDF (0.69 + 1.49 for extra voices) PDF Expert (6.99) Adobe Reader (free) speech/exams/computer-readers 12. Digital Paper Requests 2008-13 2008 2009 2010 2011 2012 2013 Number of requests 514 1,167 2,000 2,832 3,694 4,291 Number of centres 46 73 101 149 173 188 Number of candidates 204 422 675 1,069 1,327 1,677 Mean number of requests per centre 11.17 15.99 19.80 19.01 21.35 22.82 Mean number of candidates per centre 4.43 5.78 6.68 7.17 7.67 8.92 Mean number of requests per candidate 2.52 2.77 2.96 2.65 2.78 2.56 13. Digital Question Papers 2008-2013 0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000 2008 2009 2010 2011 2012 2013 Number of requests Number of candidates 14. Scottish assessment: Costs of readers/scribes 19,058 reader requests; 14,905 scribe requests ~ 36,000 individual exams ~ 54,000 hours Say average 20/hour for reader/scribe? = 1,080,000 Say 10/hour for invigilator? = 540,000 TOTAL = 1.62min Scotland last year 15. 1. Become confident with the technology and text-to- speech tool they will be using in the exam. - Use PDFs of past papers to gain experience on how to navigate the papers - Use TTS in class 2. Centre applies for digital paper 1. In Scotland via the online AAR system. 2. In the rest of the UK through the Modified Paper Route 3. Digital exam paper arrives. 1. In Scotland on CD with papers 2. In the rest of the UK exam boards differ but most allow for download 1 hour before exam from secure website How do schools and candidates use a computer reader? 16. Resolving issues with use of digital papers & text-to-speech 17. 1. Voice quality Problem Solution many of the students commented that the synthetic voice was of poor quality and was difficult to understand Nisbet, P.D., Aitken, S., Shearer, N. (2004) Trial of External Papers in Accessible PDF for Candidates with Additional Support Needs. Downloads/Reports/ 1. Scottish Heather voice licenced for use by Scottish Schools (2007). 2. Scottish Stuart voice developed and licenced for Scottish schools (2011). 18. 2. Pronunciation Problem Solution Text to speech doesnt read every word accurately, 1928 would be read as one thousand nine hundred and twenty-eight. Words are mispronounced, in particular names and places, which can affect the smooth reading of longer passages. Words containing an apostrophe are not read correctly. Nisbet, P.D. (2010) SQA Digital Papers 2010 Report. 1. 2011 and 2012 papers (154 papers; 2,044 pages; 235,205 words!) analysed for mispronunciations 2. 308 terms identified. 3. CereProc updated the voices to fix the pronunciation. 4. Or use a TTS tool with a pronunciation editor. 19. 3. Unreadable image text Problem Solution Some text is an image and cannot be read by the computer: 1. SQA desktop publishers now replace image text with text boxes with selectable, readable text. 2. By 2012, analysis showed that almost all image text elements were readable. 3. 2014 UKAAF guidance on accessible PDF exam papers 20. Impact on practice 21. Reliant on Readers? Stuck with Scribes? or Independent with ICT?