∂

What works…and who listens?Encouraging the experimental evidence base in education and the social sciences

RCTs in the Social Sciences

9th Annual Conference

Professor Carole Torgerson

School of Education

Durham University

carole.torgerson@durham.ac.uk

September 11th 2014

∂

∂

∂

First known RCT

[Walters, 1931]

“Five seniors, each of whom had a good scholarship record, pleasing personality, excellent health and fine social environment, were chosen to act as personnel counselors for the members of the freshman class, who at the end of the first eight weeks of school happened to be delinquent in scholarship in the School of Mechanical Engineering in I929-30. The 220 delinquent freshmen were divided into two groups by random sampling.” [italics added]

∂

US legislation: Institute of EducationSciences

“Scientifically valid educational evaluation employs experimental designs using random assignment, when feasible, and other research methodologies that allow for the strongest possible causal inferences when random assignment is not feasible.” [page 5][italics added]

∂

Education Endowment Foundation (EEF)

“...all EEF-funded projects are independently and rigorously evaluated...The impact of projects on attainment will be evaluated where possible, using randomised controlled trials.” [italics added]

∂

Every Child CountsThe independent evaluation

[Torgerson et al, 2013]

∂

Distribution of schools

∂

Early reading policy

∂

Two alternative designs

Systematic review of different methods to teach reading[Torgerson, Brooks and Hall, 2006]

Expert review of different methods to teach reading[Rose, 2006]

– Used systematic review design and methods

– Used expert review design and methods

– Included randomised trials undertaken anywhere

– Included examples of best practice, two UK-based trials and expert opinion

– Examined the quality of the included trials

– Did not assess the quality of the included studies

 

∂

House of Commons Education and Skills Committee, 18th July 2005

“…in conducting his review, Jim Rose will have the opportunity to draw on the findings of an independent systematic literature review of phonics use in the teaching and application of reading and spelling which we have commissioned from Professor Greg Brooks and Carole Torgerson. This delivers on the public commitment we made…in 2003 to publish an analysis of existing research on phonics teaching methodologies. The aim … is to identify what is known from existing literature about how effective different approaches to phonics teaching are in comparison with each other, including the specific area of analytic versus synthetic phonics.”

∂

Meta-analysis: Forest plot

Standardised mean difference

Favours Control Favours Phonics

-3.7709 0 3.77098

Study Standardised mean difference (95% CI) % Weight

Ability==0 Greaney 0.30 (-0.36, 0.95) 6.8 Lovett89 0.22 (-0.14, 0.57) 23.1 Lovett90 -0.20 (-0.85, 0.46) 6.9 Martinussen 0.46 (-0.30, 1.21) 5.2 O'Connor 0.57 (-0.59, 1.73) 2.2 Torgesen99 0.07 (-0.34, 0.48) 17.3 Torgesen01 -0.31 (-0.87, 0.24) 9.5 Umbach 2.77 ( 1.77, 3.77) 2.9

Subtotal 0.21 ( 0.01, 0.41) 73.9

Ability==1 Haskell 0.07 (-0.73, 0.87) 4.6 Johnston 0.97 ( 0.43, 1.51) 10.1 Leach 0.84 (-0.08, 1.75) 3.5 Skailand -0.17 (-0.78, 0.44) 8.0

Subtotal 0.45 ( 0.11, 0.78) 26.1

Overall 0.27 ( 0.10, 0.45) 100.0

∂

Publication bias: Funnel plot

Effect Size

∂

Publication bias: Funnel plot

Effect Size

∂

Publication bias: Funnel plot

Effect Size

Missing trials?

∂

Synthetic versus analytic phonics

∂

Policy decision

Ruth Kelly, Secretary of State for Education and Skills Times Mar.

21st 2006

Sir Jim Rose

Times Mar. 21st 2006

“The case for synthetic phonics is overwhelming.”

“I am clear that synthetic phonics should be the first strategy in teaching all children to read.”

∂

“Synthetic phonics does look promising,” says Carole Torgerson…one of the report's authors. “We found it had a moderate effect compared with whole-language approaches, but the evidence base for this conclusion was 12 relatively small trials, only one of which was UK-based. This would be an ideal time to do a national evaluation by implementing systematic synthetic phonics in some schools and not in others and then comparing the two.”

Economist Mar. 26th 2006

∂

“When the UK government recently introduced the ‘synthetic phonics’ method of teaching young children to read, they were told by Carole Torgerson…that they could easily bolster the slim evidence base by randomising which schools joined the programme first.”

Financial Times Mar.18th 2010

“In 2007 the Government introduced a new reading strategy for primary schools based on synthetic phonics, which matches sounds to groups of letters. Professor Torgerson urged ministers to start a randomised trial: the introduction of phonics would have been staggered, with schools chosen at random to start it one year or the next. Every child would have received the intervention, but it would have been possible to compare outcomes and establish whether phonics really works.”

Times Sept. 24th 2011

∂

Evidence of impact?

‘KS1 results cast doubt on phonics’

http://www.tes.co.uk/article.aspx?storycode=6021536

Percentage of pupils reaching level 2 or above

Subject 2006 2007 2008 2009 2010 2011 2012

Reading 84 84 84 84 85 85 87

Writing 81 80 80 81 81 81 83

Speaking and listening

87 87 87 87 87 87 88

∂

Study characteristics of educational trials 1990-2002

Characteristic Drug Health Education Number in each group 86 86 86 Sample size justified 59% 28% 0% Concealed randomisation 40% 8% 0% Blinded Follow-up 53% 30% 14% Use of CIs 68% 41% 1% Low Statistical Power 45% 41% 85%

Torgerson et al, British Educational Research Journal 2006, 31:6.

∂

Annual conference: Randomised controlled trials in the social sciences

∂

Thank you for listening!

Rose, J. (2006) Independent review of the teaching of early reading, London: Department for Education and Skills

Torgerson, C.J., Wiggins, A., Torgerson, D.J., Ainsworth, H., Hewitt, C. (2013) Every Child Counts: Testing policy effectiveness using a randomized controlled trial, designed, conducted and reported to CONSORT standards, Journal of Research in Mathematics Education: 15(2)

Torgerson DJ, Torgerson CJ The design of randomised trials in health, education and the social sciences. Palgrave Macmillan, Basingstoke, 2008

Torgerson, C.J., Brooks, G. & Hall, J. (2006) Literature review of phonics use. London: Department for Education and Skills

Torgerson, C. Systematic Reviews, Continuum, London and New York, 2003

Walters, J.E. (1931) The Journal of Higher Education: 2(8), 1931