Group 2 Program Evaluation

7/29/2019 Group 2 Program Evaluation

1/26

1

Jenna Hutto

Kayse Morris

Niki Taylor

Program Evaluation

FRIT 8425

Fall 2012Georgia Southern University

December 3, 2012


2/26

2

Executive Summary Page 3

Introduction to report Page 4

Focus of the evaluation Page 6

Brief overview of evaluation plan and procedures Page 8

Presentation of evaluation results Page 9

Conclusions and recommendations Page 15

Appendices Page 18


3/26

3

The purpose of this evaluation is to determine the impact of the program Read Naturally,

and its role in aiding student achievement based on oral reading fluency and comprehension.

This program evaluation was conducted at Bacon County Elementary school with ten students

from each third, fourth, and fifth grades. Currently this program is offered to students who

score below the twenty fifth percentile based on Lexile scores. In determining the

effectiveness, the following evaluation questions were posed:

Do students who use Read Naturally show significant gains in reading fluency andcomprehension?

Do teachers who implement Read Naturally see improvement in students on an academiclevel?

Data sources were used from the Read Naturally program at Bacon County Elementary in

Alma, Ga. These reports included students weekly probes based on achievement, areas of

weakness, and grade level. We evaluated CRCT scores and instructional folders to review

approaches for response to interventions. Students who were placed in the lower twenty fifth

percentile were nominated for tier three interventions and the Read Naturally, program

based on success rates. Students final recommendations for promotion were reviewed before

making a final decision on probing subjects. Questionnaires were distributed to classroom

teachers and parents of students who participated in the program.


4/26

4

Since the introduction of Read Naturally, in the fall of 2009, at Bacon Country

Elementary, the program has demonstrated success based on individual student achievement.

This program has helped many students enhance their reading capabilities. Read Naturally,

focuses on the five major points of reading: vocabulary, fluency, phonics, phonemic awareness,

and comprehension.

Read Naturally's structured intervention programs combine teacher modeling,

repeated reading, and progress monitoring three strategies that research has shown are

effective in improving students' reading proficiency. Using audio support and graphs of their

progress, students work with high-interest material at their skill level.

(http://www.readnaturally.com)

Read Naturally, also focuses on primary steps when assessing and aiding in better

reading.

1.) Select a story2.) Key Words3.) Prediction4.) Cold Timing5.) Graph the cold timing score6.) Read along7.) Practice
http://www.readnaturally.com/http://www.readnaturally.com/


5/26

5

8.) Answer the questions9.) Pass10.) Graph the hot timing score11.) Retell12.) Practice and pass the words list


6/26

6

The goal of Read Naturally, since 1991, is to help students become better readers. The

program is designed to help students increase reading fluency by using three main elements,

teacher modeling, repeated reading, and progress monitoring.

The program has been implemented into the regular school day schedule in 30 minute

sessions for 14 weeks at Bacon County Elementary School. The students selected to participate

in the program are students that receive a low score on the AimsWeb Screening or are

currently in any tier of intervention. AimsWeb is a universal screening of oral reading fluency

in which students read a passage to check for a students accuracy and determine their number

of correctly words read per minute. Students on tier intervention were returning users of the

program. Some of the students were new to the program Read Naturally.

Read Naturally is an individualized program. It allows students to progress at their own

pace through different reading levels. Students participating the program become more

independent and develop a feeling of ownership for their success. The program offers high-

interest nonfiction stories that are age-appropriate depending on the reader.

The purpose of the evaluation of the program Read Naturally is to focus on reading

accuracy, reading fluency, and reading comprehension. The following questions were used to

guide the evaluation:

1. Is Read Naturally a suitable program for students?


7/26

7

2. Do students who use Read Naturally show significant gains in reading fluency andcomprehension?

3. Do teachers who implement Read Naturally see improvement in students on anacademic level?


8/26

8

To assess the impact of Read Naturally on students at Bacon County Elementary School,

information was collected detailing student gains between cold timing scores and hot timing

scores and the perceptions of the stakeholders over a 14 week period. The information was

collected through student evaluations, student questionnaires, parent questionnaires, teacher

questionnaires, and system data analysis.

The evaluators collected qualitative data by providing evaluations and questionnaires

to students, parents, and teachers. The program system data analysis was collected to show

quantitative data. The evaluators examined the data of 60 students, 10 third graders, 10 fourth

graders, and 10 fifth graders. Again, some students were familiar with the program and some

were new to it.

Data of the 60 students was used to determine the gains between the cold and hot

timing scores. Periodic reports were collected for the 60 students. The reports presented the

students goal while working with the Read Naturally program (Appendix A,B,C). Fluency

details were also collected for each of the 60 students (Appendix A,B,C). Student evaluations

were distributed to the 60 students from third, fourth, and fifth grade (Appendix F).

Questionnaires were also distributed to the students (Appendix E) and their parents by the

evaluators (Appendix D). The data collected in the program evaluation of Read Naturally was

obtained from the 60 students at Bacon County Elementary School.


9/26

9

The counselor and RTI coordinator at our school analyze previous CRCT test scores and

reading fluency scores. Also, they meet with teachers and discuss students that need an

intervention in reading. After completing a list of which students meet the requirements for

interventions, the intervention team sends home letters explaining that the students need help

in the area of reading. The letter also explains they will be using Read Naturally, a computer

based program, for 30 minutes a day. The child has to remain in the intervention until they

meet the grade level goal. Below are the recommended goals for fluency for grades three

through five.

Below are the charts that illustrate the information for ten students from each grade

level. It depicts the starting date, ending date, starting WPM rate, ending WPM rate, stories

read, and comprehension correct.

Recommended Reading Fluency Rates in Connected Text

Grade Fall Winter Spring

3rd 79 84-93 100-114

4th 90-99 98-112 105-118

5th 105 110-118 118-128


10/26

10

After examining the third grade results, we noticed that 1 out of 10 students did not

increase in reading fluency and that 3 out of 10 students do not meet the standard of 80% on

comprehension correct. We recommended that teachers who are implementing Read Naturally

pay attention to what the students are doing to maybe cause no increase in WPM or less

percentage in comprehension. It may be due to the students not properly knowing how to use

the program or making mistakes. This may be something that can be improved with the help of

the teacher applying help or it may be that the student just does not understand and is missing

a step prior to being placed on Read Naturally. Another problem was that most of the third

grade students, with the exception of three, did not meet the recommended WPM goal for

winter which is 84-93. However, looking at their starting WPM goal, they are well below grade

level already. In the fall they should have been reading 79 words and most of them started at

either 50 or 60. Therefore, this tells me some of students still improved a tremendous amount.

3rd Grade

StudentStarting

Date

Ending

Date

Starting

Goal

(WPM)

Ending

Goal

(WPM)

Stories

Read

Comprehension

Correct

1 8/15/11 11/30/12 50 80 36 76%

2 8/15/11 11/28/12 50 100 22 101%3 8/15/11 11/30/12 50 80 41 87%

4 8/15/11 11/30/12 50 50 61 80%

5 8/15/11 11/30/12 50 70 50 83%

6 8/15/11 11/30/12 50 75 13 71%

7 8/15/11 11/30/12 65 125 52 82%

8 8/15/11 11/30/12 50 100 36 82%

9 8/15/11 11/30/12 50 60 34 70%

10 8/15/11 11/30/12 85 90 30 85%


11/26

11

When examining the fourth grade results, all of the students increased in WPM and only

three of the students did not reach the recommended goal of 80% in comprehension answered

correctly. Once again, this could be a problem that can be helped by the teacher or may be that

the student needs other intervention. However, most of the students in fourth grade did reach

the recommended reading fluency for winter which was 98-112. This shows better

improvement than third grade. Also, more of the students in fourth reached the 80% goal of

comprehension. However, there was still one student who fell short of the WPM goal and the

comprehension goal. This student may need a different type of reading intervention. Keep in

mind some students to not cooperate and learn well on computers. Therefore, this student

may need to be in some type of reading intervention class.

4th Grade

StudentStarting

Date

Ending

Date

Starting

Goal

(WPM)

Ending

Goal

(WPM)

Stories

Read

Comprehension

Correct

1 8/15/11 11/30/12 65 90 93 70%2 8/15/11 11/30/12 85 115 33 84%

3 8/15/11 11/30/12 80 100 30 84%

4 8/15/11 11/30/12 65 100 86 79%

5 8/15/11 11/30/12 85 100 72 81%

6 8/15/11 11/30/12 80 120 41 81%

7 8/15/11 11/30/12 85 130 52 82%

8 8/15/11 11/30/12 50 90 30 83%

9 8/15/11 11/30/12 65 100 92 80%

10 8/15/11 11/30/12 75 79 52 78%


12/26

12

Fifth grade results look awesome. Eight out of the ten students started well below WPM

for their grade level. However, after completing the program, eight out of the ten students

were well above the WPM goal for winter which is 110-118. Also, only three out of the ten

students had a comprehension that was below 80%. Several of the students had a large gain in

WPM after using Read Naturally and maintained an 80% or above on comprehension

questions correct. By the Spring fluency check, most of the fifth grade students should be

reading at or above average recommended WPM, which is 118-128. Four of the students are

already above the recommended WPM for Spring. The other six will have time to continue to

increase their WPM in reading fluency and hopefully reach the grade level goal.

5th Grade

StudentStarting

Date

Ending

Date

Starting

Goal

(WPM)

Ending

Goal

(WPM)

Stories

Read

Comprehension

Correct

1 8/15/11 11/30/12 50 120 30 87%

2 8/15/11 11/30/12 130 140 68 77%

3 8/15/11 11/30/12 120 130 22 87%

4 8/15/11 11/30/12 75 110 44 84%

5 8/15/11 11/30/12 100 100 77 80%

6 8/15/11 11/30/12 70 110 75 90%

7 8/15/11 11/30/12 85 120 106 77%

8 8/15/11 11/30/12 75 110 140 84%

9 8/15/11 11/30/12 80 115 57 80%

10 8/15/11 11/30/12 80 80 29 77%


13/26

13

Stakeholder Perceptions

After discussing the program with teachers, parents, students, and staff members, the

overall information indicates that Read Naturally has had a positive impact on the students

who participated. Some of the strengths they named were self-paced, easy to use/navigate,

and computer correction. These were some ofthe most important strengths. They believe

that Read Naturally being self-paced helps the students not feel intimated by others. Another

great feature is that they read on their own so other students cannot hear them or worry about

who is more advanced in the program.

The parents really liked the fact that the program was offered during the school day and

that students could use the program while at school without getting further behind on their

actual classwork. They also liked that while working individually on computers, there was also

a teacher present so if the students needed guidance. Another feature the parents really liked

about Read Naturally is that it gave the students immediate feedback on the story and words

read per minute. Also, the program allows them to read, reads to them, and allows them to

read again. It is teaching them how to read the passage, not just letting them read it alone.

Some features that the teachers and staff liked are the read- word and comprehension

questions. If while reading a student comes to a word they do not know, they can simply click

on the word and the program will tell them how to pronounce the word. Also, the

comprehension questions are multiple choice and if the student gets the question wrong, it will

not allow them to move on to the next question, it brings the story back up and allows them to

go back in the story and find the correct answer. Even though they answered incorrectly and

the question counts against them, they have to find the correct answer before they can move to


14/26

14

the next question or next step of the program. This teaches them skills of how to read back in

the story to find an answer.

The students were also asked what they thought about Read Naturally and how it has

benefited them in reading. The answers speak for themselves. Some students said I can read

better and I can read faster now. The students who really struggle in reading and want to

increase their reading speed and accuracy were overjoyed working on the program that it

helped them succeed. The only downfall was that some students who are on it very frequently

said that it does get boring because you are doing the same thing each day even if you are

reading different stories.

Program limitations cited by this group were concerned with student ability and

staffing concerns. More students could benefit from this program but without the staff to allow

the students to participate in the Read Naturally program, some of the students are not chosen

because they dont quite make the requirements for the program. Also, some students just do

not have the ability to use the program and actually increase in words per minute (fluency)

and comprehension.

Overall, the program definitely has more positives than negatives. There are more

strengths in the program data and the surveys from teachers, parents, and students than

downfalls in the program. This program may not be perfect and may not help every student

succeed in reading faster and more accurately, but it is a great start to help a child improve

their reading skills. Read Naturally focuses on increasing fluency and comprehension and at

Bacon County Elementary School it has done that in many ways. The program has more than

proved its effectiveness to our staff, parents, and students.


15/26

15

Based upon the evaluation of the program data relating to student needs and gains of the

current program Read Naturally, this evaluation study concludes:

Program Strengths:

The teachers manual is well-organized with clear objectives linking research with practice,plus detailed information for proper implementation and effective use of the program.

Students receive explanations as to why they perform various activitiescold timing,reading along, and practice readings.

Progress monitoring drives instructional practice, helps with instructional decisions, andalerts teachers to reading problems.

Comprehension questions and the retell step are included with every story, remindingstudents that their ultimate goal is to understand what they read.

Students are instructionally engaged in the act of reading and their time on task is high. Students work in a self-directed manner, freeing the teacher to work with other students.

Teacher assistance is required for the pass step.

Timed reading and graphs of results can be very motivating for students. Most students increase in WPM over the time period while using Read Naturally.


16/26

16

Program Weaknesses:

Lack of ample time and staff to productively implement Read Naturally as recommended. As seen in the results, it does not always work on every student, there are exceptions. Amount of comprehension questions answer correctly is not as high at least 80% for every

child.

Recommendations:

Work with scheduling at the school to allow more time for students to work on ReadNaturally each week.

Work individually with the students who have not increased in WPM to see what they maybe doing or what their problem may be.

Try to help the students who are not increasing to at least 75% on comprehensionquestions.

Therefore, for the most part, Read Naturally is a very effective program. As with every program

though, it has its weaknesses. For the most part, the school needs to get a different schedule

and try to incorporate the students using Read Naturally more during the week. Also, the

school and teachers need to figure out what individual students are not increasing in WPM are

struggling with. It may be that they do not fully understand the program or are having trouble

navigating, especially if they were not taught properly. Also, the students who do not have a

comprehension of at least 80%, which is 12 students out of the 30, may have some issues also.

The staff members who are with the students when they are working on Read Naturally may

have to work closer one-on-one and watch to see what the students are missing. As always,


17/26

17

there are some students who need further instruction or help than others. Therefore, the lower

comprehension and no increase in WPM may be a problem that can also be improved.


18/26

18

Appendix A3rd Grade Results Letter and Fluency Graph

Student 1


19/26

19


20/26

20

Appendix B

4th Grade Results Letter and Fluency Graph

Student 1


21/26

21


22/26

22

Appendix C

5th Grade Results Letter and Fluency Graph

Student 1


23/26

23


24/26

24

Appendix D

Parent Questionnaire


25/26

25

Appendix E

Student Questionnaire

1.) Do you enjoy reading?

2.) Do you believe Read Naturally, has helped yourreading accuracy and fluency?

3.) How often do you read?

4.) When do you read best?

5.) How has Read Naturally, helped you?


26/26

Appendix F

Student Evaluation

Documents

Group 2 Program Evaluation