AC Eval 2010-12. - Web viewImprove students’ reading comprehension. ... strategies. Create volunteer ... comprehension, test-taking skills, content-area reading and writing, word

Lake County Office of Education

AmeriCorps Tutoring and School Readiness Programs

2009-11 Evaluation ReportYear 2

September 2011

Robert YoungLake County Office of Education

1152 South Main StreetLakeport, CA 95453

(707) 279-1269

1



[email protected]

Introduction:

This report summarizes 2009-2011 evaluation data for Lake County Office of

Education’s AmeriCorps program funded jointly by the Corporation for National Service and our

community partners.

The program is designed to accomplish these goals:

1. Improve students’ reading fluency

2. Improve students’ reading comprehension

3. Ensure members develop an understanding of tutoring strategies

4. Create volunteer management and volunteer opportunities

This report provides highlights from the evaluation findings for the 2009-10 and 2010-11

years.

2

mailto:[email protected]



Methodology

A collaborative team effort commenced in September 2009 to design a comprehensive

and asset-focused evaluation consisting of both process and outcome evaluation components.

The evaluation team included the Lake County Office of Education (LCOE) AmeriCorps

Program Director, other staff from LCOE, and our partner sites.

Table 1Evaluation Plan

Number Completed 2009-10

Number Completed

2010-11Process Data Collection Components Member Pre/Post Evaluation Improvement (2pts increase out of 5). 30 of 32 41 of 46 Outcome Data Collection Components Matched Assessments (matched pre/post of target) 358 of 350 470 of 400

The Lake AmeriCorps program continued to see great success in the systems and

processes that were put into place during the past years’ grants. Delivering curriculum and

services, as well as collecting outcome and output data was the focus in years one and two.

Adding more qualitative data through classroom observations, tutee interviews, and interviews

3



with site supervisors was an important addition to the “big picture” of how services benefit the

tutee and partner sites. The collection of this qualitative data helps support the findings behind

the numbers collected and personalizes the program.

Program Description: Member Development

The Lake County AmeriCorps Program (LCAC) had a 100% enrollment in both 2009-10

and 2010-11. Member Retention rate did not meet the 90% target expectations in either year,

though it did increase, and maintained that increase from the past two program years.

2008-2011 Retention Rates

Year#

Enrolled # CompletedRetention

Rate2007-08 37 25 65%2008-09 36 29 73%2009-10 38 32 84%2010-11 51 43 84%

LCAC program has continued to seek input from the CLASP partnership, members

(through evaluations), program staff, and CaliforniaVolunteers (CV) on how to improve

retention. The LCAC program continues to work with CV and our partner sites on strategies to

increase our member retention.

Member development and training completed on schedule. The program’s goal of 15%

training fell just short in both years due to the lack of retention and a bizarre amount of snow and

4



ice in the spring of 2011. Overall, members showed positive improvements over the pre/post

member evaluations.

Member Evaluation Improvement

Year#

Enrolled# Pre/Posts Completed

% of Members

with Improved

Post Scores2009-10 38 32 81%2010-11 51 46 89%

Program Description: Assessments

Gray Oral Reading Test (GORT): measures oral reading rate, accuracy, and

comprehension. Information about rate and accuracy is important because children who read

slowly take longer to complete assignments, less likely to understand, and remember less of what

they have read. The assessment is research-based and proven effective.

California Literacy and Reading Project Kindergarten Assessment (CLRP): The CLRP

measures K-2nd grade students’ mastery of letter identification, letter sounds, blending sounds,

and print concepts. The assessment is research-based and proven effective.

Desired Results Developmental Profile (DRDP)- The DRDP measures preschool

student’s Literacy and Language development. This is the only assessment that our members do

not implement. Office of Education staff administer this assessment to all LCOE preschoolers

5



and share the target group results with LCAC program. The assessment is research-based and

proven effective.

Program Description: Curriculum

REWARDS: Rewards is a research-based reading and writing intervention program. It is

an effective intervention in general and special education, summer school, and after-school

programs. The curriculum focuses on decoding, fluency, vocabulary, comprehension, test-taking

skills, content-area reading and writing, word choice, as well as sentence writing and revision.

Phonics for Reading: The research-based Phonics for Reading curriculum delivers direct

instruction in phonics and increasing fluency. This three-level program builds on learning. First

Level focuses on short vowels, consonants, consonant blends, and digraphs. Second Level

progresses with vowel combinations, r-controlled vowel sounds, common endings, and CVCe

words. Third Level expands concepts with vowel/letter combinations, common prefixes and

suffixes, minor consonant sounds for c and g, and minor vowel sound combinations

Findings:

The GORT, CLRP and DRDP assessments were administered to a selected group of Lake

County students before service started during the fall of 2009 and 2010. These students were

also given a mid assessment during February and a post assessment in May. The instruments are

6



research-based assessments that provide a reliable benchmark on the fluency and/or

comprehension of these students. Targeted to pre-12th students who are struggling in their

Language Arts/English class, performed below average on state tests, or recommended by site

staff, the assessments enables LCAC to collect and analyze valuable data for identifying areas of

literacy strengths and weakness in students. They are designed to be part of a comprehensive

data-driven decision making process to help guide the development of more effective tutoring

program.

Delivery of Services:

2009-2010

Tutors spent over 20,000 hours of direct tutoring service at 31 sites during the 2009-10

school year serving 393 students. Of the 393 tutees that were pre-assessed, 358 assessed 1.0

functional age/grade levels behind the student’s actual grade level. Tutees receiving 15 or more

hours of service demonstrated a slight increase in score on the GORT. This increase was not

statistically significant improvement over their peers receiving five or more hours of service.

Analysis of these data on a pre-to-post participation basis demonstrated that statistically

significant improvements were made on the overall assessment instrument, with the average

change in scores listed below:

7



2009/10 Literacy Tutoring Demographic Information of Participants

Characteristic# of Students

Tutored

% of Students in Grade

Level

AVG. % CHANGE- Fluency

AVG. % CHANGE- Comprehension (GSRT)

Grade level Preschool 176 38.9% 2.55 Not MeasuredElementary (K-5) 190 42.1% 1.51 1.78Middle (6-8) 27 6.0% 1.82 2.05High School (9-10) 59 13.0% 1.29 2.4

452 100.0% 1.91 1.94Total

*The probability that pre-to-post change was due to random chance, values less than .05 are generally considered statistically significant.

The Lake County AmeriCorps Literacy Program was successfully implemented at all

sites. All grade levels showed statistically significant improvements. The target goal of reaching

350 tutees each year was met. The participants were equally divided by gender. The division of

8



participants by gender was nearly equal. The majority of students tutored were in grades

Preschool-5.

2010-11

Tutors spent over 27,000 hours of direct tutoring service at 33 sites during the 2010-11

school year serving 633 students. Of the 633 tutees that were pre-assessed, 470 assessed 1.0

grade levels behind their functional grade level. On the post assessments, 188 of 203 (92.6%) of

2nd-12th grade tutees receiving twelve or more hours of service increased their score by at least

1.0 grade levels and 146 students raised their functional grade level score to their actual grade

level; 157 of 213 (73.7%) of preschool students who received 12 hours of tutoring services

moved from “Developing” to “Building” level on the DRDP-r Literature and Language modules;

and 47/54 (87%) of Kindergarten-1st grade students who received 12 hours of tutoring services

increased functional grade level in fluency by 1.0 years as measured by CRLP- CA Assessment.

Tutees receiving 15 or more hours of service demonstrated a slight increase in score on the

GORT. This increase was not a statistically significant improvement over their peers receiving

five or more hours of service.

9



2010/11 Literacy Tutoring Demographic Information of Participants

Characteristic

# of Students Tutored %

AVG. % CHANGE- Fluency

To Age/Grade Level

Grade level Preschool 213 45.3% 2.75 naElementary (K-5) 201 42.8% 1.49 136Middle (6-8) 26 5.5% 1.34 9High School (9-10) 30 6.4% 1.4 1

470 100.0% 1.75 146Total*The probability that pre-to-post change was due to random chance, values less than .05 are generally considered statistically significant.

**Did not use the GSRT

The Lake County AmeriCorps Literacy Program was successfully implemented at all

sites. All grade levels showed statistically significant improvements. The target goal of reaching

400 tutees each year was met. The participants were equally divided by gender. The division of

10



participants by gender was nearly equal. The majority of students tutored were in grades

Preschool-5.

Barriers:

Completion of the evaluation tools. Due to students moving or dropping the program the

completion of the pre/post assessments is difficult, if not non-existent in certain settings

(i.e.- the community school tutees regularly go “missing” and/or transition between the

community school and Juvenile Hall).

The physical locale of tutoring. Some tutors provide service in a specified classroom

with a Reading Specialist while others tutor in the library or in the classroom (open

buildings with distractions).

Implementation with fidelity. The program performs check-ins and site visits to insure

the tutoring services are being provided with fidelity. Obtaining buy-in from the site

supervisors ensures day-to-day fidelity. 2010-11, site supervisors are aware and

somewhat knowledgeable of LCAC curriculum and assessments.

Member Retention. Member retention has steadily improved from 2007/08 to 2010/11,

but is still below the target (90%). Member evaluations stated the program should focus

11



on restructuring the training plan (Table A) to better prepare members for service and

reduced the overall amount of traveling for trainings (was implemented in 2009-10

Recovery Program).

Limitations:

The self-reported literacy approach to reporting data had a couple problems. Members

assessed tutees and self-reported results into a local database. Nearly one in twenty tutors (5%)

in 2009-11 did not provide complete or accurate information about their tutee(s). Tutees may use

different approaches for different grades (supplements fused into the curriculum), but this

information was not collected. Supplemental and other intervention programs remain unknown.

This analysis covers only two years. Finally, the program’s evaluation was completed by the

Program Director and not an experienced evaluation team.

Summary and Recommendations

This section summarizes what we have learned from the outcome evaluations of the Lake County

AmeriCorps program.

12



Phonics for Reading highlights:

The Phonics for Reading tutees exceeded the program goal of improving “their reading

level by 1.0 grade levels on the GORT.”

Examining the results of the assessments revealed a positive increase in overall reading

performance. The GORT, which measures reading fluency, had the greatest increase.

Positive change in reading fluency and comprehension is very important because these

measured skills directly related to performance in the classroom.

Recommendations:

Due to the great success of the Phonics for Reading curriculum as well as one of the

program objectives, implement the program in every district.

Conduct focus group of Phonic for Reading participants, and their teachers, to obtain

feedback about the program and how it influenced them.

Rewards & Rewards Plus:

Rewards participants exceeded the program goals of improving “their reading level by

1.0 grade levels on the GORT.”

Examining the results of the assessments revealed a positive increase in overall reading

performance. Positive change in reading fluency and comprehension is very important

because these measured skills directly related to performance in the classroom.

13



Recommendation:

Conduct focus groups of participants, and their teachers, to obtain feedback about the

program and how it influenced them.

Closely track individuals that receive tutoring to obtain outcome data (before and after

grades, attendance, and state tests scores to establish a correlation between tutoring

academic success).

Conduct one-on-one interviews with collaborative partners.

14



2009-10 Literacy TutoringData by Grade Level

Grade

# Matched Pre/Post

Pre Only

Total

GORT to Grade / Age Level

GSRT to Grade / Age Level

Pre/K to Grade / Age Level

GORT

GSRT

Student Retention

Pre 82 35 117 76 2.55** *** 0.700 32 5 37 28 *** *** 0.861 28 3 31 16 6 1.33 *1.4 0.902 40 6 46 24 29 1.68 1.88 0.873 32 8 40 24 18 1.54 2.05 0.804 21 4 25 8 10 1.29 2.13 0.845 6 5 11 2 2 1.45 2.63 0.556 15 3 18 9 6 1.76 2.63 0.837 2 2 1 1 *2.13 *3.35 1.008 6 1 7 2 0 1.81 1.34 0.869 8 9 17 3 1 1.27 *3.27 0.47

10 4 5 9 2 1 1.48 *5 0.4411 11 13 24 1 0 1.41 *0.3 0.4612 5 4 9 0 0 1.19 0.56

Totals: 292 101 393 92 74 104 1.91 2.11 0.71

*Insignificant number of students assessed.**Students Increased an average of 2.55 developmental stages on a five point scale

15



***Kindergarten students are not able to take the GORT or

GSRT.

2010-11 Literacy TutoringData by Grade Level

Grade

# Matched Pre/Post

Left Program Total

To Grade/Age Level

GORT AVG

GORT increase >1

Student Retention

Pre 213 213 2.95** *** 1.00

0 17 4 21 17 *** *** 0.811 37 14 51 38 1.59 27 0.732 56 30 86 34 1.63 50 0.653 33 20 53 23 1.87 29 0.624 32 16 48 15 1.36 25 0.675 26 14 40 9 1.04 18 0.656 11 22 33 6 1.69 10 0.337 5 12 17 0 0.7 1 0.298 10 8 18 3 1.63 7 0.569 6 5 11 1 1.26 4 0.55

10 8 7 15 0 1.34 5 0.5311 7 4 11 0 1.13 4 0.6412 9 7 16 0 1.87 8 0.56

16



Totals: 470 163 633 146 1.43 188 0.61

*Insignificant number of students assessed.**Students Increased an average of 2.95 developmental stages on a five point scale***Kindergarten students are not able to take the GORT.

Table A: Training Schedule:

Training Hrs.

Month #1:Orientation #1- AmeriCorps Basics (8hrs)Orientation #2- curriculum, assessments, data training (8hrs)Orientation Overview & Workshops (3hrs)Team Leader Training (4hrs)Time Management Training (4hrs)*Monthly Meeting: Tutoring Strategies Committee Training (1hr)

October:Crisis Prevention Training (CPI) (8hrs)Tutoring ELL\ESL Training & Thinking Maps (4hrs)Monthly Meeting- Volunteer Recruitment & Support *Monthly Meeting: TBA Make-A-Difference Day Prep (3hrs)

Nov-May*Monthly Meeting Training- Tutor Strategies

May

17



All AmeriCorps Week (2hrs)Life After AmeriCorps (8hrs)

JuneEnd of Program Total Mandatory

88 hrs

*Monthly Meetings: Every First Friday of Month from 8:30am-10am &10am 12pm for all Program Specific Training (3.5hrs x 10)

18

Documents

AC Eval 2010-12. - Web viewImprove students’ reading comprehension. ... strategies. Create volunteer ... comprehension, test-taking skills, content-area reading and writing, word