McGraw-Hill Education SRA FLEX Literacy Research

Research

Ronald C. Martella, Ph.D.

Nancy E. Marchand-Martella, Ph.D.

confidential and proprietary

confidential and proprietary2

Chapter 6 Chapter 7 Chapters 9 & 10Chapters 4 & 5 (group)

and 11-13 (single case)

Figure 1.6. Continuum of constraint levels for different research methods. (Note: The categories of quantitative

and qualitative methods are meant to be for general guidance only since case study and naturalistic and descriptive

research methods can also fall under the category of quantitative research depending on how the data are collected and

analyzed. Also, survey research (Chapter 8), program evaluations (Chapter 14), and action research (Chapter 16) can

also have aspects of both quantitative and qualitative research methods.




Research Base:

White Papers

5










DATA

14


Learner Verification Study

15

• Main purpose to provide feedback using pre-publication version of

program.

• Imperfect implementation—full program not completed, starts and stops

throughout, changes in program and technology improvements.

• Even with focus of study, positive results were seen across diverse

settings—non-special education, special education, after-school.

• Social Validation: Teachers and students liked program and reported

effects.

• Statistically significant results seen on TOWRE-2.

• Positive results on GRADE; statistically significant effects for Grade 5

Total and Comp; statistically significant effects for Vocab at Washington

site.

• Increasing trend on DIBELS.


Background

• Designed by Department of Academic Research as an Exploratory study

• Incorporates 141 students across grades 3 through 8 attending schools in GA, SC, TN, TX, and WA

• All students performing at least two years below grade level in reading

• Demographics– Higher percentage of males (58%) than females – 65% identified as Caucasian– Over half identified with special needs, most of these SLD– 86% free/reduced lunch


Implementation

• **Study utilized pre-published version of materials **

• Staggered implementation– Late September: SC– October: GA, TN, WA– Early November: TX

• Setting– GA and TX in non-special education classrooms– TN and SC in special education classrooms – WA in an ‘after-school’ setting


Implementation, cont.

• Consultants conducted on-site visits to monitor implementation throughout the study– Strong implementation for SC, TN, and TX

• All program components; 90 – 120 minutes per day for instruction

– Slight deviations observed in GA• All program components; 90 – 120 minutes per day for instruction

– Partial implementation in WA• Digital and Print; 45 minutes – 1 hour for instruction

– Initial difficulties with technology/connectivity in WA which impacted implementation of the Digital Experience


Social Validity Questions of Interest• What are teachers’ and students’ attitudes

and perceptions of the program?– Online survey for teachers, end of study

– Student interview, end of the study

• What effect does FLEX Literacy have on students’ perceived reading skill and interest in reading?– Pre- and post-program administration of a 9 item

survey


Results

• Teacher Survey


Teacher Survey

• All teachers (100%) reported that students were engaged during the Print Experience , Digital Experience, and the Project Experience.

• All teachers (100%) reported that routines were easy to follow, students felt successful, and content standards were addressed in the Print Experience.

• All teachers (100%) indicated that the Teacher’sGuide is well-organized, students felt successful, and content standards are addressed in the Project Experience.


Teacher Survey, cont.

• Responses on items addressing the Digital Experience varied across sites and largely reflect issues with technology/connectivity that have since been corrected.

• Despite this, the majority of teachers cited the Digital Experience and the opportunity for students to work at their own pace as particular strengths of SRA FLEX Literacy.


• What four words would you use to describe SRA FLEX Literacy?

• Individual• Challenging• High-expectations• Collaborative• Intense• Flexible• Thought provoking• Life time skills• Interesting• Relevant• Deliberate• Engaging• A very different format


• 100% of teachers indicated they would recommend SRA FLEX Literacyto a colleague.

• Yes. I already have. Teachers are looking for an individualized program for students struggling in reading and this program (mainly the digital) is equivalent to a student who is sitting one on one with a reading tutor. The teacher knows exactly where the student is and 30 students can work at once on their level, rather than hiring a tutor for 30 minutes of small group once or twice a week. More students benefit from individualized instruction resulting in less being labeled with a disability.


Results

• Student Interview


How did SRA FLEX Literacy help you become a better reader?

• 87% of students reported that they felt SRA FLEX Literacy helped them become a better reader.

0

10

20

30

40

50

60

70

80

Read/PronounceWords

Read Aloud Asks Questions Other


90% of students’ responses were positive, indicating they enjoyed the program or thought it improved their reading skills.

• What would you tell other students about SRA FLEX Literacy?

0

10

20

30

40

50

60

Fun/Cool Read Better Easy/Hard DK


59%16%

7%

6%

12%

Digital Experience

Print Experience

Project Experience

Learning

DK

What is the best thing about SRA FLEX Literacy?


• The majority of students responded that they enjoyed the Project Experience.

– Roughly one-third reported that their favorite part was “working in teams”

– Other responses varied considerably, but included “researching,” “coloring/creating,” “making PowerPoint slides,” and simply, “all of it.”


Results

• Student Survey


• Short, nine-item survey administered before and after participation in FLEX

• Designed to capture student perceived reading skill and interest in reading

– I am a good reader

– I like to read at home

• Response categories ranged from Yes! to Not At All


All Students

• After participation in FLEX, more students selected Yes!, or Mostly in responding to the statements:

– I am a good reader (15 % increase)

– When reading, I can easily read all of the words (14% increase)

– When reading, I know what all of the words mean (10% increase)


By Subgroup

• I am a good reader– Grade 5 (25% increase)

• I can read as well as other students– Grade 5 (8% increase)

• When reading, I know what all of the words mean– Grade 5 (20% increase)

• I would enjoy receiving a book as a present– Grade 8 (25% increase)

• When reading, I can easily read all of the words– Grade 8 (33% increase)


Results

• Achievement Measures


Questions of Interest, cont.

• Association with basic reading, vocabulary, and comprehension skills

– Pretest and Posttest using published, norm-referenced tests

• Association with oral reading fluency skills

– Repeated administrations of the Oral Reading Fluency (ORF) subtest of the DIBELS


Quantitative Measures

• Test of Word Reading Efficiency, Second Edition (TOWRE-2)– 2 subtests: Sight Word Efficiency and Phonemic Decoding

Efficiency

• Group Reading Assessment and Diagnostic Evaluation (GRADE)– Comprehension and Vocabulary Composites along with a

Total Score

• Oral Reading Fluency (ORF) subtest from the DIBELS– One-minute reading assessment of reading fluency (words

per minute)


TOWRE-2 and GRADE

• Group administered• Contain parallel forms for pretest and posttest• High reliability/validity• Give norms for off-grade level assessment• Provide standard scores/are norm-referenced

– Allow for comparison across various tests and groups– Can be manipulated for use in statistical tests– Easily interpretable – Often utilized in the identification of students with

disabilities


A word about “statistical significance”

• Indicates whether or not there is a difference between groups simply due to chance

• Is impacted in part by sample size

– If treatment effects are small (and present), larger samples are more powerful in detecting them.

– With smaller samples, treatment effects (if present) must be relatively larger to be detected.


…and Effect Size

• Describes the magnitude of the difference• Represents the average difference between groups,

expressed in standard deviation units – For example, an effect size of .30 means “three-tenths of a

standard deviation”

• Effect size is descriptive for that particular group• Cohen’s d

– ‘Small’: .20– ‘Medium’: .50– ‘Large’: .80 and greater– For educational research, .25 is considered minimal level


Learner Verification

Study (continued)• Effect sizes are measures of magnitude—i.e., how large are the effects.

• Effect sizes were positive. Note that .25 effect size is the minimal level for

educational significance (Adams & Engelmann, 1996); effect sizes over

1.00 are relatively rare in education.

Some example results:

Grade 5

Effect size = .42 on GRADE Comp

Effect size = .38 on GRADE Total

Grade 8

Effect size = 1.49 on TOWRE-2

Tennessee

Effect size = 1.37 on TOWRE-2

Washington

Effect size = .73 on TOWRE-2

Effect size = .52 on GRADE Vocab


All Students

• Statistical significance achieved for TOWRE-2, effect size is small.

• Non-statistically significant increases for the GRADE.

• For all measures, pretest and posttest scores fall within the Low Average range.


Student Performance by Grade Level

• Grades 3, 4, 6, and 7– Increases noted

• Grade 5– Statistically significant increases for GRADE

Comprehension (d = .42, small) and GRADE Total (d = .38, small effect).

• Grade 8– Statistically significant difference for TOWRE-2 (d =

1.49, large effect)


Grade 5

• Statistical significance achieved for GRADE Total and GRADE Comp; effect size is small.

• Non-statistically significant increases for the TOWRE-2 and GRADE Vocab.

• The average comprehension score increased from the Low range to the Low Average range.


Grade 8

• Statistical significance achieved for TOWRE-2,; effect size is large.

• Stable performance for the GRADE.

• The average TOWRE-2 score increased from the Low Average range to the Average range.


Student Performance By Site

• Increases shown in Georgia, South Carolina, and Texas

• Tennessee– Statistically significant increases for the TOWRE-2 (d =

1.37, large effect)

• Washington– Statistically significant increases for the TOWRE-2 (d =

.73, medium effect) and GRADE Vocabulary (d = .52, medium effect).


Tennessee

• Statistical significance achieved for TOWRE-2; effect size is large.

• Non-statistically significant increases for the GRADE.

• The average TOWRE-2 score increased from the Low Average range to the Average range.


Washington

• Statistical significance achieved for TOWRE-2; effect size is medium.

• Statistical significance achieved for GRADE Vocabulary; effect size is medium.


SLD Category

• Statistical significance achieved for the TOWRE-2, effect size is large.

• Non-statistically significant increases are noted on the GRADE Total and GRADE Comprehension.

• Stable performance on GRADE Vocabulary.

• The average TOWRE-2 score increased from the Low range to the Low Average range.


Results

• DIBELS

• Oral Reading Fluency


Oral Reading Fluency

• General upward trend across grade levels

• The average gain in words per minute by grade level ranged from a 4% to a 59% increase

• Strongest results for grade 5


GRADE 5ORF

• At baseline, students read an average of 63 words per minute.

• Students read an average of 100 words per minute on the Spring benchmark.

• Students gained, on average, 37 words per minute. This reflects a 59% increase over baseline levels.


Results

• LEXILE Measures




Because the total number of lessons students receive may have an effect on their scores, the correlation between Lexile® growth and percentage of lessons completed for the program was calculated.

The correlation between Lexile® growth and percentage of yearly lessons completed for the program was statistically significant, r = .559, n = 69, p < .001 (two-tailed). This correlation indicates there was a statistically significant, positive relationship between Lexile® growth and the percentage of yearly lessonscompleted.


Summary

• The majority of students and teachers reported that they enjoyed using SRA FLEX Literacy, would recommend it to others, and felt successful while using it.

• There were some positive shifts in perceived reading skill and interest in reading, for several groups.

• Pre-post program differences on the TOWRE-2 were determined significant for several groups.

• Oral Reading Fluency scores show an upward trend, over time.


Washington School District Study

57

• Large study in Eastern Washington School District.

• 55%-75% free and reduced price lunch.

• Primary purpose to determine student academic gains.

• Full program implemented for entire year.

• Up to 150 students across Grades 6, 7, and 8.

• Intervention students.

• Multiple measures including Washington State MSP (Measurements of

Student Progress) test results, AIMSWeb, TOWRE-2, GRADE, in-program

assessments.

• Possible math test results on MSP.

• Compare with students who are not in intervention program but with similar

demographics.