Coaches as Critical Consumers of

Research

Statewide Coaches Meeting

Oregon Reading First CenterRachell Katz

Jeanie Mercier-Smith

April 24, 2008

Objectives

Scientifically Based Reading Research Consumer’s Guide

Scientifically Validated/ Researched Instruction What Works Clearinghouse

“Look fors” in research studies Follow up on Tim Shanahan Resources to keep us up to date on research

The Challenge in Education

No “Food and Drug Administration” for Education• Medical profession has a rigorous process for

researching drugs and surgical procedures before they can be used on people.

• In education we rely on publishers to produce core programs and interventions that will result in improvements in students’ skills.

Research Based vs Proven Effective by Research

“Until an instructional practice has been implemented, evaluated and found to produce better results than its alternatives, there is no research basis for recommending it.” (Grossen, 1996)

What are some challenges

for practitioners about this

statement?

Scientifically Based Reading Research

Reading Instruction is based on the science of reading- the five big ideas that have been identified by an extensive review of the research to be critical in reading development. National Reading Panel

Lots of programs are using the buzzwords, but may not actually teach them.

Effective instructional strategies (modeling, explicit instruction, multiple opportunities to respond, systematic error correction).

Evaluating Core Programs

1. Does it teach all the relevant essential

elements?

2. Are the design and delivery adequate

for the majority of learners?

The “Consumers Guide” provides a common metric for evaluating:

1. Scope of review and prioritization of skills2. Quality and nature of the delivery of instruction

Use the following criteria for each critical element:

= Element consistently meets/exceeds criterion.

= Element partially meets/exceeds criterion.

= Element does not satisfy criterion.

When evaluating individual elements, slash ( / ) the respective circle that representsyour rating (e.g., ).

Examining Program Content

Examining Scope of Review & Prioritization The reading program’s scope and sequence should provide

evidence of breadth and depth of coverage on essential skills.

High Priority Items in KindergartenHigh Priority Items

Phonemic Awareness Instruction (5) ____ ____ ____

Letter-Sound Association Instruction (3) ____ ____ ____

Decoding Instruction (5) ____ ____ ____

Irregular Words Instruction (1) ____ ____ ____

Vocabulary Instruction (3) ____ ____ ____

Listening Comprehension Instruction (4) ____ ____ ____

High Priority Items: Grade 1 Phonics Instruction

High Priority Items — Phonics Instruction

EvidenceRating Criterion Initial

InstructionWeek

______Week

______1. Progresses systematically from simple word types (e.g.,consonant-vowel-consonant) and word lengths (e.g., numberof phonemes) and word complexity (e.g., phonemes in theword, position of blends, stop sounds) to more complexwords. (ss) [NRP, pg. 2-132]2. Models instruction at each of the fundamental stages (e.g.,letter-sound correspondences, blending, reading wholewords). (w) and (ss)3. Provides teacher-guided practice in controlled word listsand connected text in which students can apply their newlylearned skills successfully. (w)4. Includes repeated opportunities to read words in contextsin which students can apply their knowledge of letter-soundcorrespondences. (w) and (ss) [NRP, pg. 3-28]5. Uses decodable text based on specific phonics lessons inthe early part of the first grade as an intervening stepbetween explicit skill acquisition and the students' ability toread quality trade books. Decodable texts should contain thephonics elements and sight words that students have beentaught. (w) and (ss)

Scientifically Researched/ Validated Interventions

Studies that have carefully controlled implementation of practices or programs to show that the program has resulted in an increase of students skills when compared to another instructional approach.

“Research Shows…”

NCITE Tool: Evaluating Trustworthiness in Educational Research That is Used to Support a Change in Teaching or School Practice.

Is the statement the result of a single study that was conducted with 40 students?

Is the statement the result of a research review? Lots of research converging on the same conclusion Meta-analysis

Mathematica Policy Research Inc.

Findings that are not causal often are stated as causal

Mathematica Policy Research Inc.

The correlation prompts a causal recommendation

But..

The study showed a correlation

It did not measure the extent to which reducing class size will reduce expulsions

Mathematica Policy Research Inc.

What does “effective” mean?

40

60

80

0

20

40

60

80

100

Program 1 Program 2 Program 3

High School Completion Rate

Program 3 appears to be most effective

Mathematica Policy Research Inc.

What would have happened without the program matters

40

60

80

20

50

80

0

20

40

60

80

100

Program 1 Program 2 Program 3

High School Completion Rate Control Group

Program 1 appears to be most effective

Mathematica Policy Research Inc.

Observations

Showing causality is hard

Common for findings that are not causal to be stated as causal

Syntheses help to see a bigger picture but often present selective findings

What Works Clearinghouse http://ies.ed.gov/ncee/wwc/

“What’s Been Well Researched Clearinghouse” Review of the research that has been conducted

on various educational issues Only reading programs involved in rigorous

research studies of the highest quality were given a positive rating. Therefore a strong program that has not been tested

in a high quality research study is not going to receive an endorsement by WWC.

WWC Ratings Improvement Index- difference between the percentile rank of the

average student in the intervention group and the percentile rank of the average student in the comparison group. (range of -50 to +50 with positive numbers indicating results favorable to the intervention group)

Evidence Rating- considers four factors Quality of the research design Statistical significance of the findings Size of the difference between participants in the intervention and

comparison conditions Consistency in the findings across studies

Extent of Evidence- how much evidence was used to determine the intervention rating, focusing on the number and size of the studies.

Two possible categories: 1. small 2. medium to large

Purpose of What Works Clearinghouse

Implementation and feasibility of implementing programs is not the focus

Focus is on reviewing the research that has been conducted to prove effectiveness of programs

“Look Fors” in Research Studies

Random assignment to conditions Equality of groups at start No confounds ( i.e. teachers with more experience teach

intervention group, practicum students teach control group) Limitations Type of population Length of the Study Others?

____________________________________________ ____________________________________________ ____________________________________________

Practice

Identify one potential problem with this research study. Discuss with a partner. Within a school, 30 students are randomly

assigned either to a teacher who uses a new reading intervention or a teacher who uses the standard curriculum. The outcome used in the analysis is the spring score on a standardized reading test.

More Practice Identify one potential problem with this research

study. Discuss with a partner.

The teachers identified 31 children whose mean score on the fall Metropolitan Achievement Test was at about the 10th percentile. Twelve children who did not have conflicts with other requested enrichment classes were scheduled for the fall reading intervention class (intervention plus tutoring group), and seven children were scheduled for the spring class (intervention only group). The remaining 12 children made up a control group.

Be a Critical Consumer of StrategiesExample

Oregon Reading First Brown Bag Presentation with Timothy Shanahan (March 17th, 2008)

Be a Critical Consumer of StrategiesExample Article referenced by Tim Shanahan in Oregon RF Brown Bag

Presentation: Effect of Difficulty Levels on Second-Grade Delayed Readers Using Dyad Reading; Journal of Educational Research (2000)

Possible take home message that I heard: Practice in frustration-level material may result in more reading gains

What assumptions could I make?Practice = Fluency Practice?Reading gains = Increased ORFReading gains = Increased Comprehension?

Let’s look at the research…

Summary/Abstract

“The authors investigated how far above a poor reader’s instructional level dyad reading should be used to promote the greatest growth in reading level, word recognition, comprehension, and rate. Fifty-one poor readers were identified and randomly assigned to 1 of 3 instructional groups: a) dyad reading at their instructional reading level; b) dyad reading 2 grades above their instructional reading level; and c) dyad reading 4 grades above their instructional reading level. For 95 days/sessions, all groups read 15 min daily during their classroom recreational reading time. They all improved with dyad reading regardless of the difficulty level of materials. Results suggest that the difficulty level of materials used for dyad reading may make a difference in student progress.”

Effect of Difficulty Levels on Second-Grade Delayed Readers Using Dyad Reading; Journal of Educational Research (2000)

Be a Critical Consumer of Strategies Is there converging evidence?

How many studies? What is the quality of the research?

For each study: What was the intervention/instructional strategies? How many participants? Who were the participants? Does the analysis make sense? What were the results? What were the limitations?

Coaches as Consumers of Research

Where can we go for a summary of independent research of programs? What Work’s Clearinghouse Association for Supervision and Curriculum Development

(ASCD) Professional Journals

Exceptional Children, The Reading TeacherOthers? _____________________________________

Other resources: National Reading Panel National Literacy Panel on ELLs Florida Center for Reading Research

http://www.centeroninstruction.org

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

Other Resources… Teaching All Students to Read

in Elementary School: A Guide for Principals [K-3]

Effective Instruction for Adolescent Struggling Readers: A Practice Brief [4-12]

Effective Instruction for Adolescent Struggling Readers: A Practice Brief

A Synopsis of Writing Next: Effective Strategies to Improve Writing of Adolescents in Middle & High Schools [4-12]

The Secondary Literacy Instruction and Intervention Guide [6-12]

More….

http://www.dww.ed.gov/media/EL/ReadingK5/TopicLevel/el_practice_guide.pdf

Oregon Reading First Website

Research summaries posted: Assessment, Coaches, Administrators

Project Data & Norms

Does phonemic decoding skill predict reading proficiency for English Learners?

Three Year Report on Oregon Reading First: Impact and Implementation

Oregon Reading First Teacher Report (2006-07)

(relation between DIBELS and SAT-10)

Steps to take when you hear “Research Shows…”

Research based or validated by a controlled study?

How many studies have supported the claim? Check for the “look fors” when reading a

study.