FAST2 Year 2 Evaluation: 2015-2016 · blended small-group academic instruction, enrichment, and youth development activities. All services were free, including bus transportation,

FAST2 Year 2

Evaluation:

2015-2016

December 15, 2016

Shira Solomon, PhD

Table of Contents

©2016 SOLOMON EVALUATION LLC FAST2 YEAR 2 EVALUATION REPORT TO FCPS

Executive Summary................................................................................................................................ 1

Introduction

Student Population.......................................................................................................................................... 2

Activities & Resources .................................................................................................................................... 3

Program Logic Model ..................................................................................................................................... 5

Evaluation Design & Methods

Performance Objectives ................................................................................................................................. 7

Evaluation Questions ...................................................................................................................................... 8

Data Sources & Analyses .............................................................................................................................. 9

Outcomes & Findings

Student Impact

Academic Progress ......................................................................................................................... 11

Attendance ........................................................................................................................................ 12

Health Insurance .............................................................................................................................. 13

Participant Expectations & Experiences

Student Satisfaction ....................................................................................................................... 14

Family Satisfaction .......................................................................................................................... 17

Family Involvement ........................................................................................................................ 18

Family Outreach ............................................................................................................................... 20

Teacher Input .................................................................................................................................... 20

CBO Partner Perceptions .............................................................................................................. 22

Conclusions ......................................................................................................................................................23

Recommendations ................................................................................................................................25


Appendix

Population by Site & Setting .....................................................................................................................26

Student Survey ................................................................................................................................................27

Family Involvement Questionnaire ..........................................................................................................28

Family End-of-Program Survey .................................................................................................................29

Teacher Survey ................................................................................................................................................30

CBO Partner Survey .......................................................................................................................................31

Family Newsletter ...........................................................................................................................................32

This report was funded by a 21st Century Community Learning Center grant (award #165613-01)

from the Maryland State Department of Education.

Questions about the report may be directed to the author: [email protected]

Nondiscrimination Statement:

Title IX of the Educational Amendments of 1972 prohibits discrimination on the basis of gender in

educational programs or activities receiving federal funds and requires equal opportunity in

admissions, athletics, counseling, access to courses, and employment policies regarding marital or

parental status of students and treatment of students. All students and employees are covered by

this law. Questions or concerns about the application of Title IX should be directed to the FCPS

Legal Services Office.

Executive Summary


FAST in FCPS

In the 2015-2016 school year, Frederick County Public Schools (FCPS) provided extended

learning opportunities to 377 elementary, middle, and high school students through 21st

Century Community Learning Center (CCLC) grants from the Maryland State Department of

Education. FCPS operated eight 21st CCLCs across three Fun Academics in the Summer Time

(FAST) programs: FAST1, FAST2, and FAST+.

FAST2, Year 2

In its second of three program years, the FAST2 program served 116 students at Hillcrest,

Lincoln, and Waverley Elementary Schools (HES, LES, and WES, respectively). Participating

students were performing below grade level academically and belonged to low-income families.

Together, these conditions put struggling students at risk for falling increasingly behind their

peers academically and for disengaging from educational opportunities in their school and

community. The FAST2 program was designed to intercept this trajectory.

FAST2 offered students in Grades 3 through 5 a five-week, full-day summer experience that

blended small-group academic instruction, enrichment, and youth development activities. All

services were free, including bus transportation, breakfast and lunch, and supervised before- and

after-program care.

Program Performance

The evaluation found that virtually all FAST2 students avoided summer learning loss in

either reading or math, and two-thirds of them maintained or improved in both subjects.

Attendance surpassed 80 percent, and the program met its objectives for student satisfaction,

family satisfaction, and health insurance.

Participant Experience

Students perceived FAST2 as a safe and supportive environment and expressed great trust

in the teachers and staff members. The program appeared to stimulate students’ interest in

participating in the types of enrichment activities that were offered by community-based partner

organizations. Families applauded the program’s frequent communications. They were pleased

with the academic support their children received and felt that the program helped their

children to like school more.

Introduction


Frederick County Public Schools (FCPS) offers an array

of out-of-school-time programs for students and

families who need extra support. Since 2003, FCPS has

served low-achieving students in high-poverty schools

through a series of 21st Century Community Learning

Center (CCLC) grants funded by the Maryland State

Department of Education. In the 2015-2016 school

year, FCPS used three such grants to administer the

Fun Academics in the Summer Time (FAST) program.

Together, the FAST1, FAST2, and FAST+ programs

provided academic support, enrichment, and youth

development to eight schools clustered by feeder

pattern. All FAST programs ran for five weeks in the

summer of 2016.1 The summer of 2016 marked the

second of three program years for the FAST2 program,

which served rising third through fifth grade students.

Student selection was independent of students’

previous participation or eligibility.2

Student Population The FAST2 population was similar to that of the three

participating schools. The program had a higher

percentage of low-income students (95 percent in

FAST2 compared to 86 percent at HES, LES, and WES

combined) and a higher percentage of Hispanic or

Latino students (61 percent compared to 55 percent). 3

1 FAST+ also included an afterschool component at its middle school. 2 In the 2015-2016 school year, FCPS began flagging FAST participants in its student information system.

However, flags do not indicate the years that students participated. 3 Data source: http://reportcard.msde.maryland.gov/Demographics

Almost all FAST2 students were

low income and nearly half were

English learners.

N=116

Eligibility for School Services

Free or Reduced-Price Meals

FARM 95%

Not FARM 5%

Special Education

IEP 16%

Not IEP 84%

English Learner

ELL 42%

Not ELL 58%

Demographics

Race/Ethnicity

Black or African American 22%

Hispanic or Latino 61%

White 4%

Other 12%

Gender

Female 52%

Male 48%

Grade Level

Grade 3 34%

Grade 4 31%

Grade 5 34%

Home Language

English 32%

Spanish 57%

Other 11%

http://reportcard.msde.maryland.gov/Demographics


3

The student populations were also similar across the three

FAST2 sites. WES had the highest percentage of Hispanic or

Latino students and LES had the lowest (77 percent compared

to 47 percent); WES also had the highest percentage of

students whose home language was Spanish and LES had the

lowest (70 percent compared to 40 percent). The student

population is disaggregated by site in the appendix.

All three FAST2 elementary school centers were held at a

single site.4 They were taught by FCPS teachers from their

home schools but mixed with students from other schools at

meals and in their afternoon enrichment groups.

Activities & Resources

FAST2 provided an array of activities and resources that

supported student learning and engagement. FCPS teachers

provided ELA and math instruction in small-group settings,

and partnerships with community-based organizations (CBOs)

provided a variety of enrichment activities to stimulate

student engagement and promote character development.

Students had variable levels of contact with the nine CBO

partners, but they had daily contact with the Boys & Girls

Club. Although all teachers and staff shared responsibility for

creating a safe and nurturing environment, Boys & Girls Club

staff members had a unique opportunity to build supportive

connections with individual students. In addition to providing

enrichment geared toward character development, Boys &

Girls Club staff members had primary responsibility for

supervising meals and transitions between activities, and were

available to support teachers and other enrichment partners.

4 Lincoln Elementary School (LES) was home to the three FAST2 centers Hillcrest (HES), LES, and Waverley

(WES). The Monocacy Elementary School center, which belonged to the FAST1 program, also took place at

LES.

Boys & Girls Club

staff members

provided a

consistent adult

presence

throughout the

program day,

which let them

build supportive

connections

with individual

students.


4

FAST2 operated four days a week for five consecutive weeks.5 Each program day was open to

students for 11 hours, with core programming occupying six hours. Students could arrive at the

Early Risers Club as early as 7:00 A.M., where they would be supervised by Boys & Girls Club staff

members. At 8:30 A.M., FCPS buses brought students to LES. After breakfast, students had two

1-1/2 hour instructional blocks, one in ELA and one in math, followed by lunch. Meals were

served free of charge to all participating students through the U.S. Department of Agriculture’s

Summer Food Service Program.

From 1:15 to 4:30 P.M., students had two blocks of enrichment activities. After-lunch enrichment

activities were led by staff from the following CBO partners: Boys & Girls Club, FC Frederick

(soccer), Gardener’s Club, YMCA Arts, YMCA Sports, Maryland Ensemble Theatre, Maryland

Science Olympiad, TeamLink, and Young Rembrandts (visual arts). FCPS buses took students

home at the end of the second enrichment period. Students could stay at their home schools in

the After the Bell Club until 6:00 P.M.

Academics and enrichment accounted for 75 percent of the core program day.

2 HOURS – Meals, Transitions, Recess

3 HOURS

Academic Instruction

3 HOURS

Enrichment Activities

3 HOURS – Early Risers/After the Bell Club

The program also provided three “FAST principals” and community liaisons, a school nurse, and

a Spanish translator/interpreter. In addition to administering medications and first aid, the nurse

reached out to families to ensure access to healthcare and to promote health education. The

FAST principals served as a resource to teachers, site leaders, and CBO partner staff, facilitated

communication between teachers and families, and supported family involvement. Community

liaisons supported registration, transportation, attendance, and family engagement.

5 FCPS is closed on Fridays in the summer.


5

FAST2 promoted family involvement in their child’s education by encouraging communication

between teachers and families and by providing formal and informal opportunities for family

participation. All program-level documents, such as surveys and invitations to participate in

family events, were translated into Spanish, and the interpreter attended family events to

facilitate communication. All teachers and administrators reaching out to families whose home

language was not English were required to use the FCPS Language Line, which provides live

interpreters. Teachers were responsible for filling out a weekly progress report as well as

reaching out to families for phone conferences. Families were asked to sign teachers’ weekly

reports and read other programs documents in students’ weekly folders.

Program Logic Model In the FAST logic model, student engagement is the linchpin between family engagement and

academic progress. Program quality affects student progress and engagement as well as the

opportunities to leverage family and community engagement. The diagram below illustrates

these relationships and the table that follows it lists the variables associated with each category

of outcomes.

FAST2 aims to improve achievement by increasing student engagement.

PROGRAM

QUALITY


6

Program quality feeds short-term and long-term outcomes.

Program Quality Indicators Short-Term Outcomes Long-Term Outcomes

Academic Progress

Recruitment strategy

Attendance policy

Choice of curriculum and assessments

Teacher preparation/instructional planning

Quality of instruction

Instructional integration of enrichment

Avoidance of summer learning loss

Improved academic skills

Timely, full, and strategic enrollment

Increased instructional time

Achievement gains in ELA and/or math

Sustained on- or above-grade level performance

in ELA and/or math

Improved performance in other academic subjects

Improved achievement trajectory

Participation in enriching out-of-school time

activities

Student Engagement

Potency of motivational strategies

Consistency of behavior management

Quality and variety of enrichment activities

Student choice of enrichment activities

Food quality and quantity

Program attendance

Academic and social self-efficacy

Positive peer and adult relationships in

program

Enduring interest in enrichment activities

Program satisfaction

School attendance

Intrinsic motivation for academic and social goals

Positive peer and adult relationships in school

Participation in enrichment activities

Cognitive and non-cognitive engagement in

school

Family Engagement

Planned opportunities for family involvement

Communication of formal and informal

opportunities for family involvement

Communication uses translation and

interpretation

Investigation of family priorities

Families feel welcome at program

Families participate in program activities

Families feel adequately informed and heard

Families feel more capable of supporting

child’s education

Families feel more confidence in child’s school

Satisfaction with academic support and

enrichment

Satisfaction with child’s self-efficacy

Satisfaction with communication and opportunities

for family involvement

Family involvement in child’s education

Family participation in school community

Community Engagement

Program-level CBO partnership

o Partnership goals reflect program outcomes

o Partner selection reflects partnership goals

o Partner assignments align with program

characteristics

Community outreach uses culturally appropriate

media and/or locations

At-school support for daily clinical needs

Student health insurance coverage

Community presence (visibility of local

languages, ethnicities, and occupations in

program)

CBOs support family outreach

Access to healthcare

Access to health resource information

Ongoing CBO availability (opportunities for

continued student involvement)

Community partnerships (continuation and/or

expansion of FCPS-CBO program partnerships)


7

Partners promote access to community

resources

Evaluation Design & Methods


The purpose of the Year Two FAST2 evaluation was to provide an independent assessment of the

program’s performance on its objectives and an exploration of its outcomes. The evaluation also

focused on developing interim measures that could inform program implementation.

Performance Objectives

The goal of the FAST2 program is to increase academic achievement among struggling students

in high-poverty schools. Specifically, it calls for raising achievement in the core content areas of

English Language Arts (ELA) and math.

PROGRAM GOAL: By September 2017, 85% of FAST2 students will improve their reading

and/or math levels as measured by FCPS assessments.

FCPS set six objectives for measuring the program’s performance in the 2015-2016 school year.

Together, these objectives reflect a multifaceted approach to supporting students’ well-being

and engagement.

PERFORMANCE OBJECTIVES:

By September 2016, 80% of participating students will maintain or improve their reading

and/or math levels as measured by local LEA summative assessments.

By September 2016, 80% of students will have health insurance.

Average program attendance will be 89%.

80% of students will be satisfied with the program, as determined by surveys.

80% of families will be satisfied with the program, as determined by surveys.

78% of families will participate in at least one activity.


9

Evaluation Questions

The Year Two FAST2 evaluation aims to answer two broad questions:

How did the program impact students?

How did participants’ experiences in the program reflect their expectations?

These questions allow the evaluation to demonstrate the program’s performance against its

objectives, explain student outcomes, and explore participant expectations and experiences.

Student Impact The first evaluation question asks whether FAST2 produced the intended student outcomes. To

answer it, we will evaluate the quantitative evidence for the program’s contribution to students’

progress in ELA and math. We will also evaluate the scope of family involvement and the

implementation of family outreach efforts.

To what extent did students make academic progress?

How did family involvement support student education?

How did the program support family involvement?

Participant Expectations & Experiences The second evaluation question explores participant satisfaction and perceptions of program

value. To answer it, we will portray student satisfaction with the program experience, family

perceptions of program benefit, and teacher perceptions of program potential.

What aspects of the program did students value and enjoy?

How did families perceive the value of their child’s participation?

How did teachers perceive the program’s current and potential value?


10

Data Sources & Analyses

The evaluation uses both quantitative and qualitative data to demonstrate performance and to

provide evidence of outcomes. Quantitative data including demographics, attendance, and

assessments, were extracted from FCPS administrative databases in Spring 2016 and Fall 2016.

The evaluator developed surveys that were administered periodically throughout the summer

program, and conducted interviews, observations, and focus groups.

Data sources included:

Standardized assessments. Standardized Global Scholar (GS) test scores in math and

reading were used to measure academic growth because they provide stable pretest and

posttest measures of math and reading achievement. In school year 2015-2016, GS

assessments were no longer being used in FCPS district-wide and the FAST2 program did

not require HES, LES, and WES to implement these assessments school-wide. FAST2

participants took the GS assessments in June and October of 2016. Gain scores were

analyzed instead of performance bands because they offer a more sensitive measure of

change, especially in small data sets. Attendance rates were compared for students who

made academic progress and students who did not.

The evaluation also examined differences in diagnostic and curriculum-based

assessments that were administered by FAST2 teachers in the first and final weeks of the

summer program. For ELA, the program used the diagnostic Fountas and Pinnell Leveled

Literacy Assessment and the curriculum-based Math Navigator assessment. Math

Navigator units were selected based on diagnostic assessments of student knowledge.

Surveys. Surveys were employed as tools for evaluation as well as participatory events,

contributing to participants’ experience of a safe and empowering environment. Surveys

of students, families, teachers, and CBO partner staff were designed by the evaluator and

administered by program staff. Paper-based surveys allowed families to offer anonymous

feedback. Student surveys were administered using Google Forms; Survey Monkey was

used for the CBO partner and teacher surveys. Central office administrators emailed the

link to the teacher survey directly to teachers. FAST2 principals managed their teachers’

administration of the student surveys on Chromebooks during class time. Family surveys

were handed out at the June orientation sessions and the end-of-summer events, as well

as sent home in students’ weekly folders.


11

FAST2 Year Two surveys included:

o a web-based student survey (end-of-summer)

o two paper-based surveys of families (start-of-summer and end-of-summer)

o a teacher survey (summer, after mid-point)

o one survey of CBO partner staff (early summer)

Focus groups. The evaluator held three, grade-level student focus groups. The focus

groups followed strict opt-out and opt-in procedures and was supervised by an FCPS

administrator. Program enrollment forms informed families of the planned focus group

and allowed them to opt their child out; students volunteered to participate and were

free to leave while the focus group was in progress. Prior to participant selection, the

names of student volunteers were checked against the list of any students whose

families had opted them out. At the beginning of the focus group, the evaluator

explained assent and established a simple hand signal for students to indicate their wish

to leave. FAST2 teachers also participated in a focus group in the summer.

Observations & interviews. The evaluator observed academic and enrichment activities,

as well as meal times and transitions between structured activities. Informal, impromptu

interviews with teachers and staff partners were prompted by the evaluator’s

observations. Interviews and observations were intended to attune the evaluation to

features of implementation that other data collection methods may not have covered.

Implementation records. Quantitative measures of family engagement outcomes

included electronic and paper-based logs of family outreach efforts on the part of

program administrators, teachers, and the school nurse. Qualitative measures of family

engagement included artifacts such as fliers advertising family events, and program

administrators’ anecdotal reports of family outreach efforts and participation in events.

Outcomes & Findings


Student Impact

Academic Progress

PERFORMANCE OBJECTIVE: 80% of participating

students will maintain or improve their reading

and/or math levels as measured by local LEA

summative assessments.

FAST2 did not meet this target. 77 percent of

students avoided summer learning loss in either

reading or math according to their GS scores in June

and September.

Although GS assessments showed the FAST2 program

falling three percentage points short of the target,

diagnostic and curriculum-based assessments showed that

virtually all students avoided summer learning loss and

most had academic gains.6 According to GS assessments,

all but one of the students who avoided summer learning

loss in reading or math also improved their scores.7

6 Paired pretest and posttest scores were available in reading or math for 105 of the 116 FAST2 students;

102 students had paired scores in reading and 98 had both scores in math.

7 62 percent of students improved their GS Reading scores, 51 percent improved their GS Math scores,

and 31 percent improved on both GS assessments.

Diagnostic and

curriculum-based

assessments

showed that

99 percent of

FAST2 students

avoided

summer

learning loss—

94 percent in

reading and 79

percent in math.


13

Using the diagnostic Fountas and Pinnell Leveled Literacy Assessment and the curriculum-based

Math Navigator assessment,8 99 percent of FAST2 students maintained or improved in at least

one subject; 67 percent of them maintained or improved in both.9 In reading, 94 percent

maintained or increased. In math, 79 percent maintained or increased.

All assessments showed that FAST2 students avoided summer learning loss.

Attendance

PERFORMANCE OBJECTIVE: Average program attendance will be 89%.

FAST2 did not meet this target. Average FAST2 attendance was 82 percent.

Students needed to attend at least 18 (17.8) out of 20 program days to achieve 89 percent

attendance. Fully half of FAST2 students (51 percent) accomplished this. Students attended an

average of 16.4 program days. Eight percent of students missed more than half the program

days, but the other 92 percent attended an average of 17.2 days. Students who met the 18-day

attendance target did not have statistically higher gains in math or reading than students who

missed more than two days.10

8 Two Math Navigator units were selected based on diagnostic assessments of student knowledge.

Students were instructed in either, “Place value and computational strategies with larger numbers” or

“Addition and subtraction with word problems to 100.” Pretest/posttest data reflect the selected content. 9 The denominator for both percentages is the 112 students with paired scores in either reading or math. 10 Independent samples t-tests of the Spring-to-Fall differences for students attending at least 18 days

and those attending less than 18 days produced a p value of 0.30 for GS Reading and 0.70 for GS Math.

99% 94% 79%77% 63% 51%

Reading OR Math Reading Math

Diagnostic &

Curriculum-Based

Assessments

Standardized

Assessments


14

Attendance was relatively stable throughout the five weeks of the summer program. Both daily

and weekly attendance peaked in the middle of the program. Program timing may have been an

obstacle to attendance, at least in the first week. FAST2 had its first week following the July 4th

holiday weekend, running Tuesday through Friday instead of Monday through Thursday. The

weekly attendance rate was lowest in this first week (77 percent) and the daily attendance was

lowest on that Friday (72 percent). The weekly attendance rate was highest in the third week of

the program (88 percent); the highest daily attendance rate (92 percent) was also in Week 3.

Daily and weekly attendance peaked in the middle of the program.

Health Insurance

PERFORMANCE OBJECTIVE: 80% of students will have health insurance.

FAST2 met this target. 89 percent of students had health insurance (84 percent medical

assistance and 5 percent private insurance).

Insurance status varied by home language.

83%

72%

78%

88%

92%

84%

80%

86%

76%

82%

T W H F M T W H M T W H M T W H M T W H

Week 1, 77% Week 2, 83% Week 3, 88% Week 4, 83% Week 5, 79%

97% 85% 85%

English Spanish Other


15

Participant Expectations & Experiences

Student Satisfaction

PERFORMANCE OBJECTIVE: 80% of students will be satisfied with the program, as

determined by surveys.

FAST2 met this target. Surveys showed that 84 percent students were satisfied with

their overall experience in the program.

The end-of-summer survey asked students to reflect on various aspects of the program, their

experience, and their perception of its impact on themselves as students. Combined responses

to the FAST2 academics, enrichment, environment, family involvement, and student attitudes

were 78 percent positive. In comparison, 84 percent of respondents said they had a good overall

experience in the program, suggesting that the whole was greater than the sum of its parts.

Students expressed great trust in FAST2 adults. The Safe and Supportive Environment topic

received the highest percentage of positive responses (91 percent) and the lowest percentage of

‘I Don’t Know’ responses (6 percent). The statement within this topic, “A staff member will help

me if I have a problem,” received the highest percentage of positive responses (94 percent) of

any survey item. Complete results are presented in the appendix.

Students appeared to have difficulty answering the Family Involvement questions. This topic

received the lowest percentage of positive responses (58 percent) as well as the highest

percentage of ‘I Don’t Know’ responses (30 percent). However, the majority of respondents

reported positive feelings about each of the three questions in the topic, with 62 percent of

respondents saying that their family was involved in their education this summer. If ‘I Don’t

Know’ responses are not counted, the percentage of positive views increases in most topic areas.

‘Yes’ responses to the combined topic areas increases from 78 to 93 percent, and 83 percent of

students had positive views of family involvement.


16

Students gave few negative responses but quite a few gave responses that were

neither positive nor negative.

84%

79%

58%

91%

81%

82%

4%

4%

12%

2%

6%

3%

12%

17%

30%

6%

13%

15%

Overall Experience

Student Attitudes

Family Involvement

Safe & Supportive Environment

Enrichment

Academics

96%

95%

83%

98%

93%

96%

96%

95%

83%

98%

93%

96%

Overall Experience

Student Attitudes

Family Involvement


Enrichment

Academics

Yes No I Don’t Know

78%

Yes

93%

Yes

Without “I Don’t Know” responses, the percentage of positive responses is higher.

Yes


17

Enduring Interest

All but two of the 82 students taking the summer survey said they would want to continue at

least one of the enrichment activities during the year. On average, students expressed an

enduring interest in four of the nine activities. Boys & Girls Club garnered the most interest of

any CBO partner, with 71 percent of students saying they would like to continue those activities,

followed by FC Frederick (soccer) at 68 percent, and TeamLink, with 55 percent.

Student Comments

Seventy percent of students gave substantive responses to the open-ended question, “How can

we make the FAST program better?” Their ideas centered around having more of the activities

they already enjoyed or additional kinds of fun activities. A number of students felt that the

program currently embodied their idea of fun:

It is already great.

I love this program. It is the best.

I would like more activities like more soccer, more arts, more gardening.

Students’ recommendations for “more fun activities” included:

Toys and games

Dance contest

Water games

Sports like basketball and football

Field trips

Eating lunch outside

We should make a party at the end of the summer.


18

Family Satisfaction

PERFORMANCE OBJECTIVE: 80% of families will be satisfied with the program, as

determined by surveys.

FAST2 met this target. Surveys showed that 92 percent families were satisfied with the

program.

The end-of-summer survey asked families to reflect on various aspects of the program and their

perception of its effect on their children. On average, 86 percent of the 63 survey responses had

positive views of all topics. Ninety-two percent of responses reflected satisfaction with the

program as a whole and with the academic component; 92 percent affirmed that after the

summer program, “My child likes school more now.” Even higher percentages of respondents

expressed satisfaction with specific communication and academic topics:

95 percent were satisfied with how often they get information about their child,11 and

93 percent were satisfied with the academic help their child has received from his/her

teachers.

Families had highly positive views of academics and the program as a whole.

11 FAST2 sites sent home a newsletter in Spanish and English, listing contact information for the program

as well as online and community-based resources for supporting their child’s education. Content was

customized to each school’s community. One center’s newsletter appears in the appendix.

92%

86%

78%

89%

87%

92%

2%

4%

7%

5%

3%

2%

7%

10%

16%

6%

10%

7%

Overall Satisfaction

Improved School Attitudes

Opportunities for

Involvement

Communication

Enrichment

Academics

86%

Yes

Yes No I Don’t Know


19

Family Involvement

PERFORMANCE OBJECTIVE: 78% of families will participate in at least one school or

community-based partner activity that helps support their child’s education.

FAST2 met this target. There were 95 instances of family participation recorded,

representing 82% of families.

The program offered families a variety of ways to actively support their child’s education.

Teacher conferences had the highest participation (33 families had phone conferences and eight

in person), followed by attendance at the June orientation sessions (34 families) and August

end-of-program events (20). Thirteen percent of families who took the end-of-summer survey

said they participated in more than one FAST2 activity.

The survey revealed that families had positive views of their opportunities for involvement in the

summer program as well as considerable ambivalence about this topic. Although 91 percent of

respondents thought that getting involved could help their child, 71 percent affirmed that the

program offered enough ways to get involved or that they felt welcome to get involved.

Additionally, 22 percent gave ‘I Don’t Know’ responses to these two questions, suggesting that

some families may not have understood the questions or may not have felt comfortable

answering them.

Family participation patterns over the summer were similar to their patterns during the

school year. Seventy-two families of FAST2 students took the Family Involvement

Questionnaire12 (40 in English, 32 in Spanish), which portrayed the respondents as active

participants in their child’s school. As in the end-of-summer survey, parent-teacher conferences

appeared to be a particular draw for families, along with orientations and individual

conversations with teacher or administrators. Nearly half (49 percent) of respondents reported

having participated in at least one of the activities listed during the school year.

12 Complete responses to the Family Involvement Questionnaire appear in the appendix.


20

Teacher conferences were the biggest draw in the school year and the summer.

Responses to the Family Involvement Questionnaire also suggested that families felt capable of

supporting their children’s education and that they were generally satisfied with their children’s

educational environment. Complete results are presented in the appendix.

Families expressed confidence in their ability to support their children’s education.

19%

32%

39%

51%

56%

58%

86%

participated in the PTA/PTO or other committee.

attended a workshop on helping kids succeed in school.

volunteered in the classroom or at my child’s school.

attended a school sports event, concert, or play.

asked to speak with a teacher, school counselor, or principal.

attended an orientation for school or after-school activities.

attended a Parent-Teacher Conference.

School & Environment

In general, . . .

people are friendly at my child’s school. 92%

Agree or

Strongly Agree

people are respectful at my child’s school.

people are caring at my child’s school.

my child has opportunities to do activities like sports and music.

my child has safe places to hang out with friends.

I think my child is getting a good education.

I think my child is being well prepared for life.

Parental Self-Efficacy

In general, I feel like . . .

I know what’s going on in my child’s school. 94%

Agree or

Strongly Agree

I know what’s going on in my child’s classes.

I understand how my child learns.

I understand what my child likes and doesn’t like about school.

I am able to help my child with school work.

I am able to help my child with their feelings.

my child’s teachers help me help my child.


21

Family Outreach FAST2 community liaisons conducted 73 percent of the family outreach, 98 percent of which

happened by phone. There were 443 documented instances of family outreach, with an average

of 3.8 outreach efforts per FAST2 student. Most of their efforts resulted in an interaction: 63

percent of their calls were answered. At least 33 percent of phone calls focused on student

attendance. Outreach efforts were relatively constant across the five weeks of the summer

program, acquiring their greatest intensity in the last two weeks.

Outreach peaked in the last two weeks of the summer program.

Teacher Input FAST2 teachers thought the program was achieving its goals, but they were critical of the

program’s contribution to family outcomes. One hundred percent of teacher survey respondents

rated the program as excellent or good on items related to curriculum and instructional spaces

and on preventing summer learning loss, improving students’ academic skills, and increasing

students’ confidence and motivation. Only 56 percent of teachers felt the program was

increasing families’ involvement in their children’s education.

0%

10%

20%

30%

40%

Week 1 Week 2 Week 3 Week 4 Week 5

14%

26%

16%

25%

19%


22

One teacher explained that having a single site for the three FAST2 centers may inhibit the

participation of families from the other two schools: “Family involvement is very difficult for

families who live {…} in a different school neighborhood. {…} Families feel more involved when

they can visit a school in their neighborhood and see their children playing and working with

other children from the neighborhood.”13

Teachers had more positive views of student outcomes than family outcomes.

Planning, Preparation & Implementation

How would you rate the FAST program on the following aspects of planning, preparation,

and implementation?

Choosing an appropriate curriculum 92% Excellent

or Good

Training teachers to use the curriculum

Providing instructional spaces that are conducive to learning

Supplying adequate instructional materials

Training teachers in FAST-specific instructional strategies

Specifying expectations for managing student behavior

Supporting teacher management of student behavior

Specifying expectations for teacher communication with families

Supporting teacher communication with families

Student Outcomes

How would you rate the FAST program on the following student outcomes?

Preventing summer learning loss

94% Excellent

or Good

Increasing student achievement

Improving students’ academic skills

Increasing students’ confidence in their learning ability

Increasing students' motivation to do well in school

Increasing students’ interest in school

Increasing students’ enjoyment of school

Increasing students’ connection to the school community

Family Outcomes

How would you rate the FAST program on the following family outcomes?

Creating a welcoming environment for families 71% Excellent

or Good

Improving communication between teachers and families

Strengthening ties between families and schools

Helping families support their child’s learning at home

Increasing families’ involvement in their child’s education

Increasing families’ involvement in the school community

13 This teacher also called out the on-site translator, who “has been a great addition to FAST this year.

{Families} feel very comfortable speaking to her and having her speak for them.”


23

CBO Partner Perceptions All CBO partner staff were invited to participate in an online survey in the second and third

weeks of the summer. The survey received 55 responses, 35 of them from Boys & Girls Club staff

members. Participants affirmed that the program provided solid support for their partnerships

and felt that their organizations were making significant contributions to student outcomes.

CBO staff felt that the program supported their partnerships.

Partner Integration

How would you rate the FAST program on integrating your organization in the following ways?

Helping you make connections to the academic curriculum

83%

Excellent

or Good

Helping you make connections with the district’s Character Counts curriculum

Providing good spaces for your activities

Explaining the district’s expectations for managing student behavior

Helping you manage student behavior consistently across FAST program sites

Explaining the expectations for communicating with teachers, FAST administrators and other

staff partners

Helping you communicate effectively with teachers, FAST administrators and other staff

partners

Contribution to Student Outcomes

How would you rate your organization on contributing to the following student outcomes?

Increasing positive interactions with peers

93%

Excellent

or Good

Increasing positive interactions with adults

Increasing student interest in enrichment activities

Increasing students’ confidence in their learning ability

Increasing student motivation to do well in school

Increasing students’ interest in school

Increasing students’ enjoyment of school

Increasing students’ connection to the school community

Contribution to Program Success

Overall, do you feel your organization has helped the FAST program achieve its goals this

summer?

96%

Yes

Conclusions


The Year Two FAST2 evaluation asked how the program as a

whole impacted students and how participants’ experiences

reflected their expectations. The evaluation examined

academic progress and attendance; the self-reported

experiences of FAST2 students and families were augmented

by the perceptions of teachers and CBO partner staff and by

the documented outreach efforts of program administrators.

To what extent did students make academic progress?

The evaluation showed that most FAST2 students made

academic progress in at least one subject. According to

standardized assessments, FAST2 fell three points short of

meeting its target, but according to diagnostic and

curriculum-based assessments, the program surpassed its

target by 19 percentage points, with virtually all students

maintaining or improving their proficiency in reading or

math. A greater share of students made progress in reading

than in math, and a majority of students maintained and/or

improved their proficiency in each subject.

How did family involvement support student education?

Family participation patterns over the summer were similar to

school year patterns. The two family surveys showed that

teacher conferences were a particular draw for families, along

with orientations and individual conversations with teacher or

administrators. A majority of students (62 percent) felt their

families were involved with their education this summer.

How did the program support family involvement? The

program provided orientations, on-site events, and individual

outreach. Teachers had phone or in-person conferences with

most families and community liaisons had hundreds of phone

conversations with families. These outreach efforts were

congruent with families’ self-reported school-year

participation patterns.

FAST2 PERFORMANCE OBJECTIVES

ACADEMIC PROGRESS: 80% of students

will maintain or improve in reading

and/or math.

Not met. Standardized assessments

showed that 77% of students maintained

or improved; however, other assessments

showed that 99% avoided learning loss

or improved in reading or math.

HEALTH INSURANCE: 80% of students

will have health insurance.

Met. 89% of students had health

insurance.

ATTENDANCE: Average program

attendance will be 89%.

Not met. Average attendance was 82%.

STUDENT SATISFACTION: 80% of

students will be satisfied with the

program.

Met. 84% of students were satisfied with

their overall program experience.

FAMILY SATISFACTION: 80% of families

will be satisfied with the program.

Met. 92% of families were satisfied with

the program.

FAMILY INVOLVEMENT: 78% of families

will participate in an activity that

supports their child’s education.

Met. 82% of families participated in a

program activity that supported their

child’s education.


25

What aspects of the program did students value and enjoy? The student survey showed that

92 percent of students enjoyed the enrichment activities. The program seems to have stimulated

enduring interest in enrichment participation: students said they would be interested in

continuing four out of the nine activities the program offered through its CBO partners.

Students also valued the safe and supportive FAST2 environment. Excluding ‘I Don’t Know’

responses, 99 percent of students had positive views of the program environment.

How did families perceive the value of their child’s participation? Families had highly

positive views of the academic benefits of the program and of the program as a whole. Ninety-

two percent of families said that after the summer program, their child liked school more.

How did teachers perceive the program’s current and potential value? FAST2 teachers

thought the program was achieving its goals, but felt that the program could do more to

promote family involvement.

In its second program year, FAST2 enhanced its efforts to engage students and families.

Prior to the summer program, professional development sessions used last year’s

evaluation to focus teachers and CBO partners on strategies for managing student

behavior and opportunities for engaging families. This focus contributed to a

collaborative process at the administrative level and an exceptionally positive and stable

atmosphere at the program site.

FAST2 has improved its data quality by aligning assessments with instructional content. It

has also expanded the types of data it collects to include school-year family involvement

and the summer implementation of family outreach efforts. For the first time, the

evaluation captured input from the majority of families and CBO partners. It observed an

increasingly systematic and robust program, which bodes well for fulfilling the program

goal of improving elementary school students’ achievement trajectory.

Recommendations


The following recommendations are intended to help FAST2 to build on the strengths of a

successful second year:

1) Set more consistent attendance guidelines. Although many students had very good

attendance, 35 percent of students missed more than one of the five program weeks and

half of them missed more than two weeks. Low attendance can be disruptive to instructional

settings and may inhibit student engagement and achievement.

2) Engage families early in the program. Re-cast teacher conferences as outreach

opportunities by scheduling them early in the summer. Most families participated in parent-

teacher conferences during the school year and more of them participated in conferences

than in any other FAST2 activity over the summer. Connecting with families early in the

program would build on momentum generated from orientation sessions and may

contribute to greater ongoing family involvement in the program and improved student

outcomes.

3) Track family participation by student. Moving toward the use of a student-centered

denominator for would allow for a more accurate and meaningful representation of family

participation in program activities that support children’s education. Adjusting the

participation denominator by family unit also allow it to be linked to student outcomes.

4) Consolidate family and community resource information. Program administrators can

collaborate with CBO partners to compile a list of community resources, including social

services and educational resources for adults and children. Listing low-cost afterschool

programs may also help students continue their participation in the enrichment and youth

development activities that sparked their interest in the summer.

5) Coordinate the dissemination of family and community resource information.

Throughout the summer, community liaisons, teachers, and the school nurse reached out to

families for various reasons. These natural points of contact can be leveraged to more

effectively disseminate information that serve the program’s family involvement and

healthcare access objectives.

Appendix


Population by Site

HES (N=56) LES (N=30) WES (N=30)

Eligibility for School Services

Free or Reduced-Price Meals

FARM 95% 90% 100%

Not FARM 5% 10% 0%

Special Education

IEP 16% 20% 10%

Not IEP 84% 80% 90%

English Learner

ELL 50% 37% 33%

Not ELL 50% 63% 67%

Demographics

Race/Ethnicity

Black or African American 18% 37% 17%

Hispanic or Latino 61% 47% 77%

White 4% 10% 0%

Other 18% 7% 7%

Gender

Female 54% 43% 57%

Male 46% 57% 43%

Grade Level

Grade 3 32% 40% 33%

Grade 4 23% 40% 37%

Grade 5 45% 20% 30%

Home Language

English 23% 60% 20%

Spanish 59% 40% 70%

Other 18% 0% 10%


28

Student Survey This survey was given in the last week of the summer program.

N Yes No I Don't

Know

Are these statements true for you?

Academics

I am making good progress in math. 73 79% 4% 16%

I am making good progress in reading. 71 77% 3% 20%

My teachers are helping me improve my academic skills. 73 88% 3% 10%

Enrichment

The afternoon activities are interesting. 69 84% 6% 10%

The afternoon activities are fun. 73 92% 1% 7%

I want to come to FAST to do afternoon activities. 70 67% 11% 21%


I feel safe at FAST this summer. 74 89% 1% 9%

I feel safe coming to FAST and going home. 72 90% 4% 6%

A staff member will help me if I have a problem. 71 94% 1% 4%

Family Involvement

My family is welcome to come to the FAST program anytime. 76 54% 14% 32%

My family is an important part of the FAST community. 71 59% 10% 31%

My family has been involved in my education this summer. 73 62% 12% 26%

Attitudes Toward School & Self

I like school more now. 72 82% 7% 11%

I am a better student now. 74 77% 1% 22%

I am a more confident student now. 70 77% 4% 19%

Overall Experience

Overall, are you having a good experience in the FAST

program this summer? 77 84% 4% 12%


29

Family Involvement Questionnaire This paper-based questionnaire was made available to adult family members in English and

Spanish as part of the student enrollment and family orientation process.

N

Strongly

Agree Agree Disagree

Strongly

Disagree

Parental Self-Efficacy

In general, I feel like . . .

I know what’s going on in my child’s school. 67 45% 46% 9% 0%

I know what’s going on in my child’s classes. 67 40% 46% 12% 1%

I understand how my child learns. 67 43% 52% 4% 0%

I understand what my child likes and doesn’t like about

school. 69 49% 48% 3% 0%

I am able to help my child with school work. 67 43% 43% 12% 1%

I am able to help my child with their feelings. 69 52% 45% 0% 3%

School & Environment

In general, . . .

my child’s teachers help me help my child. 71 58% 34% 7% 1%

people are friendly at my child’s school. 70 49% 47% 3% 1%

people are respectful at my child’s school. 68 50% 46% 3% 1%

people are caring at my child’s school. 67 52% 45% 1% 1%

my child has opportunities to do activities like sports and

music. 68 50% 43% 7% 0%

my child has safe places to hang out with friends. 68 47% 44% 7% 1%

I think my child is getting a good education. 68 60% 37% 3% 0%

I think my child is being well prepared for life. 66 47% 45% 8% 0%


30

Family End-of-Program Survey This paper-based questionnaire was given to adult family members in the last week of the

summer program.

N Yes No

I Don't

Know

Academics Are you satisfied . . .

with your child’s progress in math? 62 92% 2% 6%

with your child’s progress in reading? 58 90% 2% 9%

with the academic help your child has received from his/her

teachers? 58 93% 2% 5%

Enrichment Are you satisfied . . .

with the types of enrichment activities that are offered? 58 88% 3% 9%

with the instruction of enrichment activities? 56 86% 4% 11%

with your child’s experience in these activities? 60 87% 2% 12%

Communication Are you satisfied . . .

with how often you get information about your child? 59 95% 3% 2%

with the usefulness of information you get about your child? 55 87% 7% 5%

with the helpfulness of staff members? 56 84% 5% 11%

Opportunities for Involvement Are there enough ways for you to get involved? 50 68% 10% 22%

Do you feel welcome to get involved? 49 73% 6% 20%

Do you feel that getting involved can help your child? 54 91% 4% 6%

Improved School Attitudes Please think about your child at the end of the school year.

Does your child like school more now? 61 92% 3% 5%

Is your child a better student now? 57 81% 4% 16%

Is your child a more confident student now? 58 84% 5% 10%

Overall Program Satisfaction Overall, are you satisfied with the FAST program this summer? 60 92% 2% 7%


31

Teacher Survey This survey was given in the second-to-last last week of the summer program.

Excellent Good Fair Poor

Planning, Preparation & Implementation (N=17)

How would you rate the FAST program on the following aspects of planning, preparation, and implementation?

Choosing an appropriate curriculum 82% 18% 0% 0%

Training teachers to use the curriculum 41% 59% 0% 0%

Providing instructional spaces that are conducive to learning 88% 12% 0% 0%

Supplying adequate instructional materials 59% 24% 12% 6%

Training teachers in FAST-specific instructional strategies 41% 41% 12% 6%

Specifying expectations for managing student behavior 12% 82% 0% 6%

Supporting teacher management of student behavior 47% 47% 0% 6%

Specifying expectations for teacher communication with families 59% 29% 12% 0%

Supporting teacher communication with families 65% 18% 18% 0%

Student Outcomes (N=16)

How would you rate the FAST program on the following student outcomes?

Preventing summer learning loss 81% 19% 0% 0%

Increasing student achievement 56% 31% 13% 0%

Improving students’ academic skills 63% 38% 0% 0%

Increasing students’ confidence in their learning ability 81% 19% 0% 0%

Increasing students' motivation to do well in school 44% 56% 0% 0%

Increasing students’ interest in school 31% 44% 25% 0%

Increasing students’ enjoyment of school 44% 50% 6% 0%

Increasing students’ connection to the school community 27% 67% 7% 0%

Family Outcomes (N=16)

How would you rate the FAST program on the following family outcomes?

Creating a welcoming environment for families 25% 63% 13% 0%

Improving communication between teachers and families 31% 50% 19% 0%

Strengthening ties between families and schools 25% 50% 25% 0%

Helping families support their child’s learning at home 25% 38% 25% 13%

Increasing families’ involvement in their child’s education 31% 25% 31% 13%

Increasing families’ involvement in the school community 25% 38% 25% 13%

Overall Program Success

Overall, do you think the FAST program is achieving its goals this summer? 100% Yes

CBO Partner Survey


32

This survey was given in the second and third weeks of the summer program.

What community-based organization do you represent? N=62 Boys & Girls Club 63% FC Frederick 11% Maryland Science Olympiad 8% Young Rembrandts 8% Maryland Ensemble Theatre 3% TeamLink 3% YMCA Arts 2% YMCA Sports 2%

N Excellent Good Fair Poor

Partner Integration

How would you rate the FAST program on integrating your organization in the following ways?

Helping you make connections to the academic curriculum 57 30% 53% 14% 4%

Helping you make connections with the district’s Character

Counts curriculum 57 35% 46% 19% 0%

Providing good spaces for your activities 58 53% 29% 17% 0%

Explaining the district’s expectations for managing student

behavior 57 42% 47% 11% 0%

Helping you manage student behavior consistently across

FAST program sites 57 35% 47% 16% 2%

Explaining the expectations for communicating with

teachers, FAST administrators and other staff partners 57 35% 49% 12% 4%

Helping you communicate effectively with teachers, FAST

administrators and other staff partners 57 39% 42% 18% 2%

Contribution to Student Outcomes How would you rate your organization on contributing to the following student outcomes?

Increasing positive interactions with peers 55 60% 38% 2% 0%

Increasing positive interactions with adults 55 49% 45% 5% 0%

Increasing student interest in enrichment activities 55 51% 45% 2% 2%

Increasing students’ confidence in their learning ability 55 53% 44% 2% 2%

Increasing student motivation to do well in school 54 30% 63% 6% 2%

Increasing students’ interest in school 54 31% 59% 7% 2%

Increasing students’ enjoyment of school 55 33% 55% 13% 0%

Increasing students’ connection to the school community 55 33% 56% 11% 0%

N Yes No I Don't Know Overall, do you feel your organization has helped the FAST

program achieve its goals this summer? 54 96% 0% 4%


33

Family Newsletter This newsletter was customized for the three FAST2 school communities and distributed to

families in English and Spanish.

!Bienvenido a Waverley FAST 2! Boletín Informativo 2016

Estamos muy contentos de que cada estudiante y sus

familias sean parte de Académicos de la Diversión en el

Verano (FAST por sus siglas en Inglés).

La meta del programa de verano (FAST) es reducir

la pérdida de aprendizaje durante el verano para los

estudiantes viviendo en la pobreza y aumentar sus

oportunidades de participación en programas de

enriquecimiento.

*¡¡Habilidades aprendidas esta semana!! *La Asistencia es Importante* Los estudiantes que asistan a FAST todos los días,

tendrán la oportunidad de ganar premios. Por favor

contáctenos si su hijo/hija necesita transporte..

Habilidades de 3er Grado: 1. Lectura y Comprensión de Textos a nivel de grado.

2. Entendiendo y usando el valor de posición para

resolver todas las operaciones (+, -, x, ÷)

Habilidades de 4º Grado: 1. Redondeando números a la decena y centena más

cercana usando una recta númerica.

2. Leyendo fluidamente y comprendiendo al nivel de

grado.

*Personal de FAST y sus Números de Telefóno*

Directora: Nikki Musselman 240-236-3900 / 240-236-2721 Enlace Comunitario: Laura Rottmund 240-236-3900 / 240-236-2721 Maestras: Pepper Bowins 240-236-2721 Lana Dayhoff 240-236-2721 Angie Decker 240-236-2721 Tammy Sander 240-236-2721 Jenna Stephen 240-236-2721

Habilidades de 5º Grado: 1. Entendiendo y usando el valor de posición para

resolver todas las operaciones (+, -, x, ÷)

2. Leyendo fluidamente y comprendiendo textos nivel

de grado.

Actividades Comunitarias Series de Conciertos en el Baker Park Domingos por la

noche, 7pm Julio & Agosto Desafío de Lectura de Verano 2016 El programa de lectura de verano de FCPL opera de Mayo

1o – Agosto 31st y es para niños, adolescentes, y adultos.

¡Lea con su familia y gane premios! Puede hacerlo Online o

en su biblioteca local.. http://www.fcpl.org/programs/summer-reading 301-600-1630 Habilidades de Lectura de Verano: Programas Online Programas Individualizados que incluyen participación en la

instrucción online, lectura independiente en buenos libros, y

un calificado y alentador maestro (4 años - grado 12)

Fonética, Comprensión, Fluidez, Habilidades de Estudio de

Libros de Texto, Lectura Veloz, Amor por la Lectura,

Llamar al 1-800-964-9974 Lunes-Viernes 8am-9pm y Sábado de 9am-4pm

Casa Abierta La Casa Abierta de Noche de Regreso a Clases de

Waverley tendrá lugar el 18 de Agosto de 4:00pm -

6:00pm. ¡Vengan a Waverley a conocer a sus

maestros y a recibir sus MOCHILAS! Recuerde,

TODOS los útiles escolares les serán proveídos a

su hijo/hija el primer día de clases.

*Programa de Servicio de Comida en el Verano

Almuerzo Gratis para muchachos hasta los 18

años de edad Lunes – Viernes

● Hill Street Park 12:30-1:30

● Stonegate Park on Andover Lane 12:00-12:45

● Carver Community Center on Lee St. 12:00-

12:30

● Carrollton Park on Center St & Prospect Blvd

12:00-12:45

http://www.fcpl.org/programs/summer-reading

http://www.fcpl.org/programs/summer-reading


34

Documents

FAST2 Year 2 Evaluation: 2015-2016 · blended small-group academic instruction, enrichment, and youth development activities. All services were free, including bus transportation,