Evaluation Report Attachments #1-5 · FFY 2017 Annual Report Evaluation Attachments Evaluation Report Attachment # 1 Process and Outcome Evaluation: Tracking Policy, Systems,

Evaluation Report Attachments #1-5:

UC CalFresh Annual Report

FFY 2017

FFY 2017 Annual Report Evaluation Attachments

Evaluation Report Attachment # 1

Process and Outcome Evaluation: Tracking Policy, Systems, and Environmental Change

(PSE) Activities through Program Evaluation and Reporting System (PEARS)

Project: UC CalFresh SNAP-Ed Activities in 31 California Counties


Process and Outcome Evaluation: Smarter Lunchrooms Movement (SLM) Self-Assessment

Scorecard Evaluation

Project: Smarter Lunchrooms Movement of California Collaborative (SLM of CA)


Formative and Outcome Evaluation: Measuring UC CalFresh’s Direct Education Success via

Adult & Youth Evaluation Tools



Process, Outcome and Impact Evaluation: Shaping Healthy Choices Program (SHCP)

Evaluation Report FFY2017

Submitted by Center for Nutrition in Schools – UC Davis

Project: SHCP Pilot Counties – Butte, Placer, Riverside San Joaquin, Santa Barbara, Sutter,

Tulare, Yuba


Process Evaluation: UC CalFresh Youth Engagement Initiative Mid-Point Assessment Report

Submitted by Public Health Institute Center for Wellness and Nutrition


Page 2 of 61

Evaluation Report Attachment #1:

Process and Outcome Evaluation: Tracking Policy, Systems, and Environmental Change (PSE) Activities through Program Evaluation and Reporting System (PEARS)


Total Cost of Evaluation:

The evaluation data collection, reporting and analysis costs are integrated into the overall cost of UC CalFresh program implementation. The cost of maintaining, further developing and supporting California State Implementing Agencies to use the Program Evaluation and Reporting System (PEARS) is additionally covered through a contract between the California Department of Social Services (CDSS) and Kansas State University’s (KSU) Office of Educational Innovation and Evaluation (OEIE).

Project Goals:

This evaluation most directly assesses the following California SNAP-Ed State Level Goal:

Goal 4: Increase access to and/or appeal of healthy dietary choices and decrease access to

and/or appeal of unhealthy dietary choices where people eat, live, learn, work, play, or shop.

Goal 5: Increase access to and/or appeal of physical activity opportunities for SNAP-Ed eligible

populations.

However, these environmental changes are also intended to impact the additional individual-level SNAP-Ed State Level Goals:

Goal 1: Increase Consumption of Healthy Foods and Beverages and Decrease Consumption of

Unhealthy Foods and Beverages

Goal 2: Increase Physical Activity

Evaluation Design:

All UC CalFresh County programs used PEARS to report Policy, Systems and Environmental (PSE) activity implemented during FFY2017. As described in the PEARS PSE module, “at the environmental level, the focus of evaluation is not on measuring changes in individuals, but rather changes in settings or venues in low-income areas where nutrition education is [typically] provided”. PSE activities were reported for a total of 353 sites/organizations. Program implementation and data collection occurred over the entire year; however, reporting was limited to August (when the revised PSE module became available) through October 6, 2017.

Results:

UC CalFresh worked on policy, systems, and environmental (PSE) changes in 353 total sites/

organizations (all stages of PSE work). When examining only those who reported implementing or

maintaining changes, a total of 965 PSE changes were adopted reaching 134,549 SNAP-Ed-eligible

individuals in 303 sites/organizations across all settings (see table 1). These results reflect PSE changes

adopted in 29 of the 31 UC CalFresh counties. Progress towards integrating PSE efforts is currently

underway in San Francisco and Trinity Counties. San Francisco conducted needs assessments at nine

sites with plans for changes in Federal Fiscal Year (FFY) 2018, and Trinity County developed plans to

pilot new PSE efforts in FFY 2018.

Page 3 of 61

Table 1: Number of Sites with One or More PSE Change Adopted and Reach by County

COUNTY Schools (K-12,

elementary, middle, & high)

Early care & education

(ECE)

Before/ afterschool programs

Other* COUNTY TOTALS

COUNTY REACH

Alameda 0 27 0 0 27 1,893

Amador (cluster) 0 0 0 1 1 11

Calaveras 6 0 0 0 6 4,000

El Dorado 7 2 1 1 11 2,858

Tuolumne 3 0 0 2 5 2,297

Butte (cluster) 7 0 0 1 8 5,745

Colusa 3 0 0 0 3 1,410

Glenn 3 0 0 0 3 3,075

Sutter 5 0 10 0 15 6,977

Yuba 12 0 8 0 20 15,310

Fresno (cluster) 4 2 0 1 7 1,733

Madera 4 0 0 0 4 956

Imperial 6 9 1 3 19 2,174

Kern 0 1 0 2 3 68

Kings 10 1 9 1 21 16,221

Tulare 9 0 0 0 9 5,067

Merced (cluster) 3 0 0 0 3 13,615

Stanislaus 5 0 4 1 10 2,166

Placer (cluster) 4 0 0 0 4 877

Nevada 1 0 0 0 1 270

Riverside 14 9 4 1 28 20,649

San Francisco 0 0 0 0 0 0

San Mateo 8 1 5 0 14 2,880

Santa Clara 14 10 2 0 26 7,292

San Joaquin 4 7 1 3 15 2,423

San Luis Obispo 7 0 0 0 7 1,199

Santa Barbara 6 0 0 0 6 7,592

Shasta (cluster) 15 0 0 0 15 4,989

Tehama 0 2 0 0 2 322

Trinity 0 0 0 0 0 0

Yolo 4 2 4 0 10 480

STATE TOTALS 164 73 49 17 303

STATE REACH 123,613 4,781 4,646 1,509 134,549 *Other includes community centers, community organizations, family resource centers, farmers' markets, food banks and pantries, public housing, residential treatment centers, shelters, and other places people live/live nearby, learn, and/or work.

PSE changes or associated promotion were most frequently adopted in three settings: schools (K-12,

elementary, middle, and high), early care and education (ECE), and before/after school programs. Over

half (56%) of the PSE changes adopted were related to nutrition, approximately one-third (34%)

addressed physical activity (PA), and 10% were associated with both nutrition and PA changes. In total,

236 sites/organizations in 28 counties made at least one nutrition supports PSE change and 187

sites/organizations in 27 counties made at least one PSE change related to PA or reduced sedentary

behavior.

Page 4 of 61

Programs, Packages and Initiatives Supporting PSEs

UC CalFresh county programs incorporated several programs, packages, and initiatives to support their

PSEs in an effort to build comprehensive and mutually reinforcing interventions (see table 2). Counties

reported engaging in initiatives related to edible gardens most often reaching 129 sites (43%) in 23

counties. The Coordinated Approach to Child Health (CATCH) and Smarter Lunchrooms Movement

(SLM) were also commonly utilized to support PSE changes and reported at nearly one-third of sites.

County programs reported School Wellness Policy efforts at one-quarter (24%) of their PSE sites and

Farm-to-School/Fork work at one in five (20%) sites. Playground stencils were reported at 19 sites (6%).

Table 2 provides a complete list of the programs, packages, and initiatives used to support UC CalFresh

PSE efforts.

Table 2: Number of Sites and Counties Delivering Programs, Packages, and Initiatives to Support

PSEs

Which of the following programs, packages or initiatives were used as part of the PSE efforts?*

# of PSE sites (n=303)

% of PSE sites

# of counties

Edible Gardens (eat, work, or learn in garden) 129 43% 23

Coordinated Approach to Child Health (CATCH) 96 32% 18

Smarter Lunchrooms Movement (SLM) 95 31% 23

School/ECE Wellness Policy – updating and/or implementing 74 24% 17

Farm-to-School/Fork 61 20% 10

Harvest of the Month (HOTM) 51 17% 11

Rethink Your Drink (RYD) 35 12% 6

Sports Play Active Recreation for Kids (SPARK) 22 7% 3

Playground Stencils 19 6% 8

Nutrition and Physical Activity Self-Assessment for Child Care (NAP SACC)

18 6% 1

Shaping Healthy Choices Program 14 5% 5

Safe Routes to School (SRTS) 5 2% 3

Children’s Power Play Campaign 4 1% 1

Youth Participatory Action Research Projects (YPAR) 3 1% 3

California Thursdays 3 1% 2

EatFresh.org 3 1% 3

Let’s Move! Active Schools 1 0% 1

Cooking with Kids, Inc. 1 0% 1

Contra Costa Child Care Council’s Best Practices (CHOICES) 1 0% 1

Healthy Behaviors Initiative (HBI) 1 0% 1

*Only includes those PSEs in the implementation and maintenance stages

Complementary Strategies Implemented with PSEs

In FFY 2017, the PEARS PSE data provide UC CalFresh with the ability to examine the layering of

complementary strategies to achieve multi-component interventions. In total, 294 (97%) of the 303 total

sites/organizations incorporated at least one complementary strategy for an average of 2.2

complementary strategies per site/organization. As shown in table 3, evidence-based education and staff

training on continuous program and policy implementation were the two complementary approaches

most often incorporated as part of UC CalFresh multi-component PSE interventions. A total of 43 (14%)

of the 303 sites/organizations implemented all four complementary strategies in coordination with their

PSE efforts to enhance the likelihood of impact and sustainability. With the continued use of PEARS in

Page 5 of 61

future years, UC CalFresh will be able to examine progression over time in the implementation of multi-

component PSE interventions.

Table 3: Number of Sites Implementing Complementary Strategies to Support PSEs

Of Those Implementing or Maintaining PSE Changes Adopted

Number (%) of Sites (n=303)

Evidence-based education 264 (87%)

Marketing 119 (39%)

Parent/ community involvement 98 (32%)

Staff training 173 (57%)

Implemented all four complementary strategies 43 (14%)

*Only includes those PSEs in the implementation and maintenance stages

PSE Changes Adopted in the Top Three Settings

The following tables display the PSE changes adopted related to nutrition, PA and both nutrition and PA

in the school, ECE, and before/ afterschool settings, representing the vast majority (94%) of the PSE site

activities reported by UC CalFresh county programs in PEARS.

School Setting

In the school setting (see table 4), the most common nutrition changes related to the Smarter

Lunchrooms Movement (SLM), farm-to-school, and edible gardens. SLM changes commonly adopted

include improving the layout/display of foods to encourage healthier selections, ensuring meal service

staff encourage healthy selections, and some that overlapped with farm-to-school efforts, such as

partnering with food service to conduct cafeteria taste tests to improve appeal and acceptability of fruits

and/or vegetables, and expanding the use of fresh or local produce (farm-to-table). In addition, many

schools worked to maintain edible school gardens, used the gardens for nutrition education, and

incorporated the garden produce into the meals and snacks served to students. These nutrition-related

PSE changes were supported by a wide range of complementary SNAP-Ed programs, packages, and

initiatives including SLM (at 87 sites), farm-to-school (at 32 sites), Harvest of the Month (at 20 sites), and

Shaping Healthy Choices Program (at 13 sites) (see table 5).

The PA changes reported most often included improving the quality of physical education (PE),

integration of classroom PA breaks, and increasing access to structured PA. Improvements in PE and

PA programming were most commonly achieved through incorporating CATCH activities (30 schools).

Two of these schools reported using both CATCH and SPARK. In addition, three of the seven schools

that reported playground stencils also reported integrating CATCH programming.

Improvements in school wellness policies and monitoring progress were the changes that most often

related to both nutrition and PA. Out of the 49 sites working to update and/or implement school wellness

policies, over two-thirds (69%) reported SLM and a number of other programs, packages, and initiatives

were reported. Table 5 below displays all of the programs, package, and initiatives reported in the school

setting. These findings illustrate the layering of UC CalFresh direct education with one or more PSE

approaches to achieve more comprehensive nutrition programming and PA integration at schools to help

facilitate healthy lifestyle changes among SNAP-Ed eligible students and their families.

Page 6 of 61

Table 4: School Settings - Policy, Systems, Environmental (PSE) Changes

164 school sites/organizations (146 schools and 18 school districts) in 25 counties adopted a total of 442 PSE changes with an estimated reach to 123,613 students

Nutrition (n=304) Improved layout or display of meal foods/beverages to encourage healthier selections (e.g. Smarter

Lunchrooms) (65)

Initiated or expanded use of the garden for nutrition education (49)

Reinvigorated or expanded an existing edible garden (35)

Initiated or expanded farm-to-table/use of fresh or local produce (23)

Ensured meal service staff encourage healthy selections (18)

In partnership with food service program, conducted cafeteria taste tests to improve appeal and acceptability of fruits and/or vegetables (18)

Improved enrollment procedures to increase NSLBP meal participation including universal breakfast/ lunch (12)

Initiated or improved point-of-purchase, decision, and/or distribution prompts (including information intended to influence choices at the point of decision) (10)

Initiated or expanded use of onsite garden produce for meals/snacks provided onsite (9)

Improved menus/recipes (variety, quality, offering lighter fares) (9)

Established or improved salad bar (8)

Initiated or expanded implementation of guidelines on use of food/beverages in the classroom, as rewards, or during celebrations or educational programs (7)

Improved free water access, taste, quality, smell, or temperature (6)

Established a new edible garden (5)

Initiated or expanded implementation of guidelines for healthier snack options or healthier competitive food/beverage options (5)

Initiated or expanded implementation of guidelines for meal foods/beverages (5)

Improved or expanded cafeteria/dining/serving areas or facilities (4)

Improve appeal, layout or display of snack or competitive foods to encourage healthier selections (3)

Initiated or expanded a mechanism for distributing onsite garden produce to families or communities (2)

Improved or expanded kitchen/food preparation facilities (2)

Improved food purchasing/donation specifications or vendor agreements towards healthier food(s)/beverages (1)

Increasing access to fresh fruits and vegetables at farmers' markets (1)

Initiated or enhanced limits on marketing/promotion of less healthy options (1)

Eliminated or reduced amount of competitive foods/beverages (1)

Initiated or improved menu labeling, e.g. calories, fat, sodium, added sugar counts (1)

Initiated, improved or expanded healthy fundraisers (1)

Established lunch recycle, solid waste and composting program (1)

Outdoor school mural focused on garden education (1)

Rules for foods served in meetings or in classrooms (1)

Physical Activity (n=99) Improved quality of physical education (23)

Initiated or expanded incorporation of physical activity into the school day or during classroom-based instruction (not recess/free play or PE) (20)

Increased access/opportunities for structured physical activity before/after school (15)

Increased or improved opportunities for physical activity during recess (13)

Improved or expanded physical activity facilities, equipment, structures (7)

Initiated or improved playground markings/stencils to encourage physical activity (7)

Increased school days/time spent in physical education (4)

Incorporating physical activity into the school day or during classroom-based instruction (3)

Increased access or safety of walking or bicycling paths (2)

Initiated or expanded restrictions on use of physical activity as punishment (2)

Improved quality of structured physical activity (1)

Initiated new or expanded access to facilities for after-hours recreation or shared use (1)

New or improved access to structured physical activity programs (1)

Both Nutrition and Physical Activity (n=39) Established or improved food/beverage, physical activity and/or wellness-related policies (20)

Established or improved a monitoring or reporting system for food/beverage, physical activity, and/or wellness related policy (19)

*Summary statistics include only those PSEs in the implementation and maintenance stages.

Page 7 of 61

Table 5: School Settings - Programs, Packages, and Initiatives Reported in PEARS

Which of the following programs, packages, or initiatives were used as part of the PSE efforts?

# of school sites (n=164)

% of school sites

Smarter Lunchrooms Movement (SLM) 87 53%

School Wellness Policy – updating and/or implementing 49 30%

Farm-to-School 32 20%

Coordinated Approach to Child Health (CATCH) 30 18%

Harvest of the Month (HOTM) 20 12%

Shaping Healthy Choices Program 13 8%

Rethink Your Drink (RYD) 11 7%

Playground Stencils 7 4%

Other, please specify: primarily edible gardens 7 4%

Safe Routes to School (SRTS) 5 3%

Children’s Power Play Campaign 4 2%

California Thursdays 3 2%

Youth Participatory Action Research Projects (YPAR) 2 1%

Sports Play Active Recreation for Kids (SPARK) 2 1%

EatFresh.org 2 1%

Let’s Move! Active Schools 1 1%

Cooking with Kids, Inc. 1 1%

Early Care and Education (ECE) Setting

In the ECE setting (see table 6), the most common nutrition changes related to edible gardens, working

to implement wellness policies, and farm-to-table efforts. In addition to maintaining edible gardens, many

ECE sites used their gardens for nutrition education and incorporated the garden produce into the meals

and snacks served to preschool children. Edible gardens were supported by Harvest of the Month and

farm-to-preschool programming delivered at 27 (37%) of the ECE sites (see table 7). Five of these ECE

sites also integrated SLM. Wellness policy related work at ECE sites tended to center around

implementing guidelines for healthier snacks, meals, and beverages as well as guidelines for healthy

classroom rewards and celebrations. Out of the 25 sites working to update and/or implement ECE

wellness policies, nearly three-quarters (72%) reported using the Nutrition and Physical Activity Self-

Assessment for Child Care (NAP SACC) and 80% integrated Rethink Your Drink programming to support

their efforts.

The PA changes reported most often included increasing the access to and quality of structured PA,

improving PA facilities/equipment/structures, and expanding opportunities for unstructured PA time/free

play. Improvements in structured PA were implemented most frequently through the incorporation of

CATCH ECE activities at two-thirds (67%) of the ECE program sites. Of the 12 ECE sites that reported

playground stencils, three-quarters (75%) also reported integrating CATCH ECE programming.

Improvements in school wellness policies and monitoring progress were the changes that most often

related to both nutrition and PA. Table 7 displays all of the programs, package and initiatives reported in

the ECE setting. UC CalFresh county programs layer direct education with PSE strategies at ECE sites

to achieve both comprehensive nutrition and PA programming that supports lifestyle changes among

SNAP-Ed-eligible preschool children and their families.

Page 8 of 61

Table 6: Early Care and Education (ECE) Settings - Policy, Systems, Environmental (PSE)

Changes

73 ECE sites in 12 counties adopted a total of 391 PSE changes with an estimated reach to 4,781 children

Nutrition (n=191) Reinvigorated or expanded an existing edible garden (31)


Initiated or expanded implementation of guidelines for healthier snack options or healthier competitive food/beverage options (28)

Initiated or expanded farm-to-table/use of fresh or local produce (23)

Initiated or expanded implementation of guidelines for meal foods/beverages (22)

Initiated or expanded implementation of guidelines on use of food/beverages in the classroom, as rewards, or during celebrations or educational programs (22)


Initiated or improved point-of-purchase, decision, and/or distribution prompts (including information intended to influence choices at the point of decision) (6)

Established a new edible garden (5)

Improved layout or display of meal foods/beverages to encourage healthier selections (e.g. Smarter Lunchrooms) (3)

Improve appeal, layout or display of snack or competitive foods to encourage healthier selections (2)


Physical Activity (n=148) Improved quality of structured physical activity (43)

Increased access/opportunities for structured physical activity on-site (40)


Increased or improved opportunities for unstructured physical activity time/free play (23)

Initiated or improved playground markings/stencils to encourage physical activity (10)

Initiated or expanded restrictions on use of physical activity as punishment (1)

Both Nutrition and Physical Activity (n=52) Established or improved a monitoring or reporting system for food/beverage, physical activity, and/or

wellness related policy (27)

Established or improved food/beverage, physical activity and/or wellness-related policies (24)

Established new or strengthened limits on entertainment screen time (1)


Table 7: Early Care and Education (ECE) Settings - Programs, Packages, and Initiatives Reported

in PEARS


# of ECE sites

(n=164)

% of ECE sites

Coordinated Approach to Child Health (CATCH) Early Childhood 49 67%

Farm-to-Preschool 27 37%

Early Childhood Education Wellness Policy – updating and/or implementing 25 34%



Nutrition and Physical Activity Self-Assessment for Child Care (NAP SACC) 18 25%

Playground Stencils 12 16%


Contra Costa Child Care Council’s Best Practices (CHOICES) 1 1%

Before/Afterschool Setting

In the before/afterschool setting (see table 8), the most common nutrition changes related to maintaining

edible gardens and using the garden for nutrition education. Edible gardens were supported by Harvest

of the Month (HOTM) and farm-to-fork programming delivered at six (12%) of the before/afterschool

sites. The PA changes reported most often included increasing the access to and quality of structured

PA through the incorporation of CATCH and SPARK activities at nearly three-quarters (73%) of the

Page 9 of 61

before/afterschool program sites. Before/afterschool programs typically selected one structured PA

program, either CATCH or SPARK (both reported at ~40% of sites), with only three sites using CATCH

and SPARK. Improving wellness-related policies of before/afterschool programs was the change most

often related to both nutrition and PA. Table 9 displays all of the programs, package and initiatives

reported in the before/afterschool setting.

Table 8: Before/Afterschool Program Settings - Policy, Systems, Environmental (PSE) Changes

49 before/ afterschool sites in 11 counties adopted a total of 99 PSE changes with an estimated reach to 4,646 children

Nutrition (n=22) Reinvigorated or expanded an existing edible garden (13)


Began offering a federal food program (CACFP, TEFAP, summer meals, etc.) (1)



Physical Activity (n=76) Improved quality of structured physical activity (33)

Increased access/opportunities for structured physical activity on-site (28)


Increased or improved opportunities for unstructured physical activity time/free play (5)

Increase in school days/time spent in physical education (1)

Initiated new or expanded access to facilities for after-hours recreation or shared use (1)

New or improved access to structured physical activity programs (1)

Both Nutrition and Physical Activity (n=1) Established or improved food/beverage, physical activity and/or wellness-related policies (1)


Table 9: Before/Afterschool Program Settings - Programs, Packages, and Initiatives Reported in

PEARS


# of before/ afterschool sites (n=49)

% of before/ afterschool

sites

Sports Play Active Recreation for Kids (SPARK) 20 41%

Coordinated Approach to Child Health (CATCH) 19 39%


Farm-to-Fork 2 4%



Youth Participatory Action Research Projects (YPAR) 1 2%

Healthy Behaviors Initiative (HBI) 1 2%

In summary, UC CalFresh county programs reported considerable achievement in both the number of

sites implementing and maintaining PSE changes as well as the number and variety of PSE changes

adopted in FFY 2017. PSEs tended to be implemented in school, ECE, and afterschool settings mirroring

those where direct education is typically delivered and therefore primarily reached children and youth.

Key PSE accomplishments supported improvements in both nutrition and physical activity integration and

continue to be in areas the State Office has focused efforts with training and technical assistance to

strategically build capacity in SLM, CATCH, and SHCP (edible gardens, school wellness). In FFY 2018,

UC CalFresh will pursue additional support of school wellness policies efforts through a partnership

(MOU) with the Alliance for a Healthy Generation and will begin to track PSE progression using the new

PEARS reporting system. The main areas of focus will be improvements in the type and number of PSE

changes adopted across settings and the implementation of additional complementary strategies to

support PSEs.

Page 10 of 61

Description of how evaluation results will be used:

The PEARS PSE evaluation results will be used in a variety of ways. They help to characterize the scope

of the PSE related activities being undertaken by UC CalFresh county programs as well as the specific

changes being adopted and estimated SNAP-Ed eligible persons being reached. This information will be

very helpful for tracking progress toward more comprehensive programming as well as reporting on

SNAP-Ed Evaluation Framework indicators at the environmental level. Since this was the first year all UC

CalFresh counties reported through PEARS, the information and experience will also be used to continue

to improve the quality and completeness of reporting through further standardization of definitions,

ongoing technical assistance and responses to “real-world” reporting questions and scenarios. FFY2017

also marks the first time all California SNAP-Ed State Implementing Agencies (SIAs) used the same

program reporting system so the results can also be used for statewide summary, tracking and program

improvement.

Point of Contact:

Questions regarding this attachment can be directed to:

Angie Keihner, MS

UC CalFresh Nutrition Education Program

University of California Davis

Email: [email protected]

Page 11 of 61


Process and Outcome Evaluation: Smarter Lunchrooms Movement (SLM) Self-Assessment Scorecard Evaluation

Project: Smarter Lunchrooms Movement of California Collaborative (SLM of CA)


This evaluation report summarizes the Smarter Lunchrooms Movement (SLM) Self-Assessment

Scorecards collected as part of the work of the SLM of California Collaborative during the 2016-2017

School Year (SY). For UC CalFresh, evaluation costs are integrated into the overall cost of implementing

the SNAP-Ed program. However, for some collaborative members, such as the Dairy Council of

California, SLM Assessments were conducted with support from non-SNAP-Ed funding.

Project Goals:

This evaluation most directly assesses the following California SNAP-Ed State Level Goal:

Goal 4: Increase access to and/or appeal of healthy dietary choices and decrease access to

and/or appeal of unhealthy dietary choices where people eat, live, learn, work, play, or shop.

However, these environmental changes are also intended to impact the additional individual-level SNAP-Ed State Level Goal:



Evaluation Design and Data Collection:

UC CalFresh is part of the SLM of CA together with the Dairy Council of California (DCC), California

Department of Education (CDE) and California Department of Public Health (CDPH). In partnership with

the Cornell Center for Behavioral Economics in Child Nutrition Programs (BEN Center), the SLM of CA

Collaborative provides training and support to disseminate the SLM intervention in California. SLM

provides schools with low-cost and no-cost lunchroom changes that encourage students to make

healthier food choices. Lunchroom makeovers that improve the convenience, attractiveness, and

normativeness of healthy options have been shown to increase the selection and consumption of fruits

and vegetables.

Trained UC CalFresh and Dairy Council SLM Technical Assistance Providers (TAPs) report SLM

Scorecard data into a single online portal. The BEN Center’s SLM Self-Assessment Scorecard is an

environmental scan that includes 100 evidence-based strategies for food service operations, cafeteria

layout, and design. The Scorecard helps food service staff and TAPs identify specific changes for

potential improvement as well as aspects of the cafeteria operations that already reflect behavioral

economic principles. Beginning in April 2014, DCC and UC CalFresh TAPS have been entering SLM

scorecard data into an online Survey Gizmo portal. CDPH contractors have participated in various SLM

trainings and are using the SLM scorecard at school sites; however, since they are not consistently

participating as TAPs, few report SLM scorecards through the online portal.

Results:

2016-2017 School Year

Table 1 summarizes the SLM scorecard assessments conducted during the 2016-2017 School Year –

from August 1, 2016 through June 30, 2017. Two hundred and twelve SLM scorecards were reportedly

conducted at 168 schools. UCCE or UC CalFresh TAPs entered the largest number of Scorecards,

followed by Dairy Council (DCC) TAPs and “Other” which includes Orange County Department of

Education and UC San Diego (UCSD), which are both CDPH SNAP-Ed Local Implementing Agency

Page 12 of 61

Subcontractors. The UCCE schools were exclusively low-income SNAP-Ed-eligible schools where at

least half the students were eligible for the Free or Reduced Priced Meal Program based on SY16-17

data. The majority (74%) of the DCC schools were also SNAP-Ed eligible. Across schools completing the

SLM Scorecard, the potential reach is over 100,000 students.

Table 1: SLM assessments conducted 2016-2017 School Year1

Total DCC UCCE Other2

Total # SLM Scorecard assessments completed

212 88 101 23

Total # of schools assessed 168 76 71 21

Total # SNAP-Ed-eligible schools (≥ 50% Free or Reduced Price Meal)

145 55 71 19

Total # of students at schools assessed 108,182 53,681 40,762 13,739

Total # of SNAP-Ed-eligible students3 (≥ 50% Free or Reduced Price Meal)

80,038 36,641 33,099 10,298

1. From August 1, 2016 through June 30, 2017 2. Other organization includes – Orange County Department of Education (a NEOPB Local Implementing Agency) and UC San Diego Center for Community Health both SNAP-Ed-funded through the California Department of Public Health. 3. Free and Reduced Price Meal (FRPM) 2016-2017 data was used. One DCC school did not have FRPM data.

Food Service, Wellness, and US School Challenge Collaboration

As indicated in Table 2, over three quarters of the SLM assessments were completed together with Food

Service Staff and the majority of schools were receiving nutrition education. More than half (59%) of the

TAPs were also actively serving on the school or district wellness committee. Very few of the schools

where SLM assessments were conducted were already recognized by the Healthier US School

Challenge (HUSSC).

Table 2: Food Service, Nutrition Education, School Wellness and Healthier US School Challenge 2016-2017 School Year

Total DCC UCCE Other

Total # SLM Scorecard assessments completed

212 88 101 23

# and % of assessments completed with Food Service Staff

169 74 79 16

(80%) (84%) (78%) (70%)

Total # of schools assessed 168 76 71 21

# and % schools receive nutrition education in SY16-17

96 42 54 0

(57%) (55%) (76%) (0%)

# and % schools where TAPs reported that they were actively serving on a wellness

committee for this school or school district

99 32 49 18

(59%) (42%) (69%) (86%)

# of schools currently recognized by Healthier US School Challenge (HUSSC)

9 7 2 0

First and Subsequent SLM Assessments

During the 2016-2017 School Year, UC CalFresh TAPs reported completing first-time SLM assessments

at 107 schools that had a total enrollment of over 50,000 students. For the great majority of schools,

TAPs reported that the initial assessment was completed prior to any SLM strategies being adopted.

Once action plans are developed and SLM strategies adopted, TAPs are encouraged to reassess the

cafeteria. During the 2016-2017 School Year, 97 schools were assessed for their second or subsequent

time.

Page 13 of 61

Table 3: SLM Assessments conducted during 2016-2017 School Year


Total # of schools assessed for the 1st time 107 41 48 18

Of those schools assessed for the 1st time, # assessed prior to any SLM interventions being

made 103 41 44 18

Total # of students enrolled at schools assessed for the 1st time1

52,411 20,933 22,859 8,619

Total # of schools assessed for the 2nd time or later

97 45 48 4

1. Student enrollment not available for one DCC school.

Change in Scores

For schools with follow-up assessments, it is possible to assess whether there has been an improvement

in their scores. Several schools were assessed more than twice (n=18) allowing for comparison from the

first to the most recent score or from the previous to most recent score. Table 4 includes the results

comparing the most recent to the previous score. On average, the length of time between the previous

and most recent SLM assessment was 9.3 months and overall score increased by 12 points. The DCC

schools demonstrated a more dramatic improvement in assessment scores than UCCE schools. The

majority of DCC schools (38 of 44) had received Team Nutrition grants from the California Department of

Education (CDE) to implement SLM strategies compared to only five of the UCCE schools. The decline

in the average scores for the four “other” schools is notable; but it reflects the very small sample size. In

addition, for three of these four schools, the previous score was reported by a TAP from another

organization and may reflect to some degree differences by observer and over time. Comparison from

the first to the most recent score for these same four schools does demonstrate a small increase in their

overall score

Table 4: For schools with 2nd or Greater SLM Assessments during the 2016-2017 School Year


# of schools assessed for the 2nd time or later in 2016-2017 School Year1

96 45 47 4

Average most recent score (maximum score=100)

54.9 57.2 54.1 39.0

Average previous score (maximum score=100)

42.9 41.1 44.3 46.5

Average first score (maximum score=100) 42.4 40.4 44.7 37.8

Average change in score from previous to most recent

12.0 16.1 9.7 -7.5

Average length of time btw previous and most recent scorecard 2016-2017 (in months)

9.3 10.3 7.9 13.8

Schools where a/any new SLM strategy(ies) were adopted since the previous scorecard assessment.2

79 40 38 1

1. One school assessed for the 2nd time or later had no first scorecard data so it was excluded from this table. 2. 3. Missing data for this question for two DCC schools and three 'Other' schools.

Page 14 of 61

When TAPs entered follow-up assessments, they were prompted to report whether any SLM strategies

had been adopted at the school since the previous scorecard. The great majority of schools (87%)

reportedly had adopted new SLM strategies. In some cases, a response to this question is missing

(n=5) for example when a scorecard was identified as a “first” assessment; but, when combined with

previous years’ data an earlier scorecard had actually been entered for that same school.

For schools that reportedly had adopted SLM strategies, TAPs were asked to specify

the SLM Scorecard sub-section in which the strategy(ies) can be found, and

to indicate the strategy adopted from a listing of options.

TAPS reported between approximately 50-70% of the schools had adopted SLM strategies pertaining to

signage/communication, focusing on fruits and promoting vegetables and salad and approximately 30-

40% adopted strategies pertaining to entrée of the day, lunchroom atmosphere, moving more white milk,

student involvement and reimbursable meals (see Table 5). Reported less frequently were SLM

strategies pertaining to recognition and support of school food and a la arte items. TAPs also reported

schools had adopted a variety of additional strategies not specifically promoted by SLM typically related

to specific promotional strategies or cafeteria improvements e.g. having a fruit slicing station or a water

station, adding a digital menu board or TV, involving student volunteers or establishing a Student

Nutrition Action Committee/Council. See Table 5 for additional details.

Table 5: For Schools with 2nd or Greater SLM Assessments during the 2016-2017 School Year1

Total (n=92)

DCC (n=43)

UCCE (n=48)

Other (n=1)

Creating School Synergies: Signage, Priming & Communication

64 35 28 1

Focusing on Fruits 52 22 29 1

Promoting Vegetables and Salad 45 23 22 0

Entrée of the Day 36 19 16 1

Lunchroom Atmosphere 35 17 18 0

Moving More White Milk 35 20 15 0

Student Involvement 30 16 14 0

Increasing Sales Reimbursable Meals 26 14 12 0

Recognition & Support of School Food 11 3 7 1

A la Carte 8 3 5 0

Other - 16 12 4 0

Dairy Council of CA 'Other' includes - Bought mascot costume of tiger to promote healthy meals (1), Cafeteria manager presented to each classroom on how to create a reimbursable meal (1), Digital Menu Board (1), Fruit slicing stations (with fruit messages) twice a week (1), Implementation of a digital TV in the cafeteria for the purpose of nutrition education and promotion of menu items (2), Improve perception of school meals (2), Increase confidence in child nutrition staff to promote school meals (2), Increase consumption/decrease food waste (2), Placing milk in nice bucket instead of industrial tray (1), Providing a water station in the eating area for the students (1), Started breakfast program (1), Student volunteers helped serve lunch (1), and Taste Don't Waste Campaign and buttons with slogan were worn by all cafeteria staff (1).

UC CalFresh 'Other' includes - Focus on signage (1), Student Nutrition Action Committee (1), Student Nutrition Advisory Council (1), and Did not specify (1). 1.The sample size (n) does not include the five schools that were missing data for the question.

Across the 79 schools that TAPs reported had adopted at least one SLM strategy since the previous

assessment, a total of 96 specific SLM strategies were reportedly adopted (See Table 6). On average

Page 15 of 61

these schools reportedly adopted 15.4 SLM strategies. While the question focused on strategies adopted

since the previous scorecard (on average 9.3 months earlier), TAPs may have been referring to even a

longer period.

The most frequently adopted strategies by at least one fifth or 19 schools included ones related to

signage such as menu boards, posters, signage, and floor decals. Another common SLM change was

descriptive or creative naming of fruit and vegetable items. Specifically for fruits, the most common

strategies were giving age-appropriate names to daily fruit options, offering at least two types of fruits

daily, offering sliced or cut fruit daily and displaying whole fruit options in attractive bowls or baskets. The

most common vegetable-specific strategies included creative or descriptive names and the display of

name cards next to daily vegetables. Other common SLM strategies mentioned by at least a fifth of the

schools pertained to listing targeted entrees on a menu board, positioning white milk so it is the first

beverage option seen in milk coolers, decorating lunchroom equipment with decals/magnets/signage and

carrying out student surveys to inform menu development, dining space decor and promotional ideas.

The BEN Center has released an updated and shortened SLM scorecard with 60 rather than 100 items.

The new scorecard focuses on those strategies found to be especially promising or impactful. The new

scorecard also introduces some new strategies aimed at improving the acceptability of fruits and

vegetables such as offering taste tests each year and self-serving spices and seasoning for students to

add flavor to vegetables. The SLM Collaborative of California will be transitioning to this updated version

in FFY 2018. Particularly promising areas for UC CalFresh programs includes transitioning from offering

taste test opportunities in the classroom to working with food service staff and youth leaders to offer

tastings in the cafeteria especially of local produce or menu items identified by food service staff as

needing greater promotion. In addition to closer integration of SLM efforts with direct education

interventions, SLM offers excellent opportunities for youth engagement and leadership development

efforts.

Table 6: The specific SLM strategy(ies) reported adopted by schools with 2nd of Greater SLM Assessments conducted during the 2016-2017 School Year1

Total (n=92)

DCC (n=43)

UCCE (n=48)

Other (n=1)

Creating School Synergies: Signage, Priming & Communication

64 35 28 1

Menu boards featuring today's meal components are visible and readable within all service and dining areas

36 21 14 1

Posters displaying healthful foods are visible and readable within all service and dining areas

35 24 11 0

All promotional signs and posters are rotate, updated or changed at least quarterly

30 19 11 0

Signage/posers/floor decals are available to direct students toward all service areas

27 17 10 0

Signs promoting the lunchroom and featured menu items are placed in other areas of the school such as the main office, library, or gymnasium

24 12 12 0

A dedicated space/menu board is visible and readable from 5ft away within the service or dining area where students can see tomorrow's menu items

24 16 8 0

All creative and descriptive names are rotated, updated or changed at least quarterly

24 16 8 0

Dining space is branded to reflect student body or school 18 11 7 0

A monthly menu is available and provided to all student families, teachers and administrators

17 7 10 0

A monthly menu is visible and readable within the school building

17 10 7 0

Page 16 of 61


Total (n=92)

DCC (n=43)

UCCE (n=48)

Other (n=1)

A weekly "Nutritional Report Card" is provided to parents detailing what their student has purchased during the previous week

3 3 0 0

Focusing on Fruits 52 22 29 1

Daily fruit options are given creative, age-appropriate names 27 11 15 1

At least two types of fruit are available daily 19 6 13 0

Sliced or cut fruit is available daily 19 10 9 0

Whole fruit options are displayed in attractive bowls or baskets

19 13 6 0

A mixed variety of whole fruits are displayed together 18 10 8 0

Daily fruit options are available in at least two different locations on each service line

17 13 4 0

Daily fruit options are easily seen by students of average height for your school

13 6 7 0

Fruits options are not browning, bruised, or otherwise damaged

12 5 7 0

Daily fruit options are written legibly on menu boards in all service and dining areas

12 4 8 0

At least one daily fruit option is available near all registers 11 8 3 0

Fruit is available at all points of sale 7 4 3 0

Daily fruit options are bundled into all grab and go meals available to students

3 2 1 0

Promoting Vegetables and Salad 45 23 22 0

Available vegetable options have been given creative or descriptive names

21 13 8 0

All vegetable names are printed/written on name-cards or product IDs and displayed next to each vegetable daily

19 12 7 0

Daily vegetables options are available in at least two different locations on each service line

16 6 10 0

All vegetable names are written and legible on menu boards 16 7 9 0

At least two types of vegetables are available daily 13 5 8 0

Individual salads or a salad bar is available to all students 12 7 5 0

The salad bar is highly visible and located in a high traffic area

12 6 6 0

Daily vegetable options are easily seen by students of average height for your school

12 5 7 0

All vegetable names are included on the published monthly school lunch menu

12 2 10 0

A default vegetable choice is established by pre-plating a vegetable on some of the trays

11 4 7 0

Vegetables are not wilted, browning, or otherwise damaged 10 4 6 0

Self-serve salad bar utensils are smaller for croutons, dressing and other non-produce items

10 4 6 0

Page 17 of 61


Total (n=92)

DCC (n=43)

UCCE (n=48)

Other (n=1)

Self-serve salad bar utensils are at the appropriate portion size or larger for all fruits and vegetables offered

8 3 5 0

Entrée of the Day 36 19 16 1

All targeted entree names are written and legible on menu boards

23 12 10 1

Daily targeted entree options are highlighted on poster or signs

17 8 8 1

All targeted entree names are printed/written on name-cards or product IDs and displayed next to each respective entree daily

16 11 5 0

All targeted entree names are included on the published monthly school lunch menus

15 6 8 1

A daily entree option has been identified to promote as a "targeted entree" in each service area and for each designated line

14 8 5 1

Daily targeted entree is easily seen by students of average height for your school

13 8 4 1

All targeted entrees are replenished so as to appear "full" throughout meal service

10 5 5 0

Lunchroom Atmosphere 35 17 18 0

Lunchroom equipment is decorated with decals/magnets/signage etc. wherever possible

20 12 8 0

Staff is encouraged to model healthful eating behaviors to students

18 6 12 0

Students artwork is displayed in the service and/or dining areas

13 4 9 0

The dining space is used for other learning activities beyond meal service

13 6 7 0

Sneeze guards in all service areas are clean 12 5 7 0

Teachers and administrators dine in the lunchroom with students

11 4 7 0

Cleaning supplies and utensils are returned to a cleaning closet or are not visible during service and dining

10 3 7 0

Dining and service areas are clear of any non-functional equipment or tables during services

10 3 7 0

Staff smiles and greets students upon entering the service line continually throughout meal service

10 3 7 0

Trash on floors, in, or near garbage cans is removed between each lunch period

9 2 7 0

Clutter is removed from service and dining areas promptly 9 3 6 0

Trays and cutlery are within arm's reach to the students of average height for your school

9 2 7 0

Cafeteria monitors have good rapport with students and lunchroom staff

9 3 6 0

Obstacles and barriers to enter service and dining areas have been removed

8 2 6 0

Compost/recycling/tray return and garbage cans are tidied between lunch periods

7 1 6 0

Compost/recycling/tray return and garbage cans are at least 5ft away from dining students

7 3 4 0

Page 18 of 61


Total (n=92)

DCC (n=43)

UCCE (n=48)

Other (n=1)

Students who do not have a full reimbursable meal are politely prompted to select and consume a fruit or vegetable option by staff

7 3 4 0

All lights in the dining and service areas are currently functional and on

6 1 5 0

Moving More White Milk 35 20 15 0

White milk crates are placed so that they are the first beverage option seen in all designated milk coolers

21 11 10 0

White milk is placed in front of other beverages in all coolers 17 10 7 0

White milk represents 1/3 of all visible milk in the lunchroom 17 10 7 0

White milk replenished so all displays appear "full" continually throughout meal service and after each lunch period

17 7 10 0

White milk is promoted on menu boards legibly 15 9 6 0

White milk is easily seen by students of average height for your school

11 4 7 0

All beverage coolers have white milk available 9 5 4 0

White milk is available at all points of sale 9 3 6 0

White milk is bundled into all grab and go meals available to students as the default beverage

2 1 1 0

Student Involvement 30 16 14 0

Student surveys are used to inform menu development, dining space decor and promotional ideas

19 9 10 0

Student groups are involved in modeling healthful eating behaviors to others

15 8 7 0

Student groups are involved in creation of artwork promoting menu items

14 4 10 0

Students, teachers, and/or administrators announce daily meal deals or targeted items in daily announcements

9 5 4 0

Student groups are involved in the development of creative and descriptive names for menu items

7 3 4 0

Increasing Sales Reimbursable Meals 26 14 12 0

Reimbursable "Combo Meal" pairings are available and promoted daily

15 8 7 0

All reimbursable "Combo Meal" names are written and legible on menu boards

11 7 4 0

A reimbursable combo meal pairing is available daily using alternative entrees

10 7 3 0

A reimbursable meal can be created in any service area available to students

9 6 3 0

All reimbursable "Combo Meal" names are included on the published monthly school lunch menu

8 5 3 0

A reimbursable meal has been bundled into a grab and go meal available to students

7 3 4 0

Grab and go reimbursable meals are easily seen by students of average height for your school

7 3 4 0

Page 19 of 61


Total (n=92)

DCC (n=43)

UCCE (n=48)

Other (n=1)

The school offers universal free lunch 7 4 3 0

Students have the option to pre-order their lunch in the morning or earlier

7 3 4 0

The cafeteria accepts cash as a form of payment 7 5 2 0

The convenience line offers only reimbursable grab and go meals with low-fat non-flavored milk fruit and/or vegetable

6 3 3 0

Reimbursable "Combo Meal" pairings are promoted on signs or posters

6 5 1 0

Reimbursable "Combo Meal" pairings have been provided creative or descriptive age-appropriate names

4 4 0 0

Reimbursable "Combo Meal" pairing names are written/printed on name-cards, labels, or product IDs and displayed next to each respective meal daily

4 2 2 0

Grab and go reimbursable meals are available at a convenience line/speed window

3 2 1 0

The named reimbursable "Combo Meal" is promoted during the school's morning announcements

3 1 2 0

Recognition & Support of School Food 11 3 7 1

The school participates in other food program promotions such as: Farm to School, Chefs Move to Schools, Fuel Up to Play 60, etc.

10 2 7 1

A local celebrity is invited to share lunch with students 3 to 4 times a year

1 1 0 0

The school has applied or been selected for the Healthier US School Challenge

0 0 0 0

A la Carte 8 3 5 0

Students must ask to purchase a la carte items from staff members

4 1 3 0

Students must use cash to purchase a la carte items which are not reimbursable

3 2 1 0

Half portions are available for at least two dessert options 1 0 1 0

Other 16 12 4 0

1The sample size (n) does not include the five schools that were missing data for this question.


The evaluation results will be used to track the expansion of the SLM Movement in California. The results help to highlight the specific types of SLM strategies being adopted to promote students’ healthy school meal selection. The results are also useful for identifying areas for future attention. For example, UC CalFresh is using the results to follow-up with TAPs about schools that did not demonstrate an improvement in the SLM scores to explore why and to identify the most promising next steps. School specific results are also shared with county programs for their work with partners.

Point of Contact:

Questions regarding this attachment can be directed to:

Barbara MkNelly, MS UC CalFresh Nutrition Education Program – Evaluation Analyst

University of California Davis Email: [email protected]

Page 20 of 61

mailto:[email protected]


Outcome Evaluation: Measuring UC CalFresh’s Direct Education Success via Adult & Youth Evaluation Tools



The direct education evaluation costs are integrated into the overall cost of implementing the program.

Project Goals:

The FFY 2017 UC CalFresh direct education evaluation results are summarized below grouped

according to two of the statewide behavioral outcome areas with findings from the adult and youth

programs presented separately. In addition, statewide results for the SNAP-Ed Evaluation Framework

individual-level priority indicators are summarized in Template C.

California Statewide Goals and Objectives

Goal 1: Increase Consumption of Healthy Foods and Beverages and Decrease Consumption of Unhealthy Foods and Beverages

By September 30, 2019, improve the dietary quality of meals and snacks consumed by the

SNAP-Ed eligible population (consistent with the current Dietary Guidelines for Americans) by

3%.

By September 30, 2019, increase consumption of fruits and vegetables among the SNAP-Ed

eligible population by 3%.

By September 30, 2019, decrease consumption of added sugar from food and beverages

among the SNAP-Ed eligible population by 3%.

Goal 3: Improve Food Resource Management

Annually at least 80% percent of surveyed SNAP-Ed adult participants report improving at least

one food resource management behavior (such as reading labels, shopping with a list and

comparing prices to maximize use of limited resources to support a healthy diet).

Annually at least 30% of surveyed SNAP-Ed adult participants report increased food security

(defined as not running out of food at the end of the month.

Evaluation Design:

All UC CalFresh UCCE county programs conduct outcome evaluation of their education activities

applying the UC CalFresh Evaluation Taskforce recommended evaluation tool(s) and Specific,

Measurable, Agreed Upon, Realistic, and Time-Based (SMART) objectives assigned to the specific

curricula or workshop topic they are implementing. Depending on the direct education curricula and/or

education format used, UC CalFresh county programs complete the recommended evaluation tool and

enter the data throughout the year into statewide data entry portals. The UC CalFresh State Office

analyzes and shares back with the county programs county and aggregated state-level evaluation results

relative to the SMART objectives.

Page 21 of 61

Adult Program – FFY 2017 Evaluation Results

For the healthy foods and beverages objectives, adult evaluation results from three different evaluation

tools are summarized below.

Intent to Change Survey (ITC) for short duration series or single lessons/workshops: Due

to participant availability and venue specific opportunities and/or limitations, nutrition education

delivery varies for adult audiences. For this reason, a simple evaluation instrument was

developed for use with a single lesson of a multi-lesson series or one-time workshops. The ITC

questionnaire is brief (only three questions) and focuses on a single current and intended future

behavior as well as an open question to solicit feedback about the workshop/lesson. While not

ideal for measuring outcomes, the ITCs do provide useful information about participants’ current

behaviors and their readiness to change. In addition, asking participants to reflect upon and

report their intentions regarding a specific behavior can help to “nudge” or encourage participants

to take action.

Adult Taste Testing Tool: Many of the lessons and workshops include food tastings in an effort

to increase exposure, willingness, and ultimately consumption of healthy foods such as

vegetables. The Adult Taste Testing Tool was developed to capture adults’ response to the taste

test.

Food Behavior Checklist (FBC) pre/post surveys for multi-session curricula: The visually

enhanced FBC pre/post survey is the evaluation tool used for outcome evaluation of several

curricula e.g. Eating Smart Being Active, Eat Smart Live Strong, Learn at Home, and MyPlate for

My Family. Of the three evaluation tools, this is the one best suited to measuring reported

behavior change.

Intent to Change (ITC) – Increasing Consumption of Healthy Foods and Beverages

The tables below summarize the 9,033 UC CalFresh ITC survey results across 24 counties in terms of:

the number of participants surveyed,

the percentage not currently practicing the desirable healthy eating behavior and

of those not currently practicing the desirable behavior, the percentage who reported the intention

to practice the desirable behavior “more often” in the next week.

Table 1. Intent To Change for Behaviors Related to Increasing the Consumption of Healthy Foods and Beverages

Current Behavior Questions Intended Behavior Questions*

During the past week, did you eat [or drink]…

N % not practicing the desirable

behavior

Within the next week, how often will you eat

[or drink]…

N % reporting intent to do it “More

Often”

A breakfast that included at least 3 food groups?

81 36% A breakfast that includes at least 3 food groups?

29 72%

Whole grains or whole grain products every day?

312 35% Whole grains or whole grain products?

109 77%

Lower-fat milk products at least 2 times a day?

782 30% Lower-fat milk products? 231 58%

Foods from all 5 food groups each day?

1,853 24% Foods from all 5 food groups each day?

450 82%

Fruit at least 2 times a day? 302 22% Fruit? 66 70%

More than 1 kind of vegetable each day?

952 20% More than 1 kind of vegetable each day?

190 80%

Choose a smaller amount of food or beverages at least 1

time?

311 15% Choose a smaller amount of food or beverages?

47 49%

NOTES: *Of those not currently practicing desirable behavior.

Page 22 of 61

Of the seven ITC healthy eating behaviors (Table 1), eating a breakfast that includes three food groups

in the past week had the highest percentage of participants (36%) reporting that they did not currently

practice the desirable behavior. Of those participants not already practicing the healthy eating behaviors,

approximately 50-80% reported the intention to practice the desirable behavior “more often” in the next

week. For several of ITC topics, the majority of participants report already practicing the desirable

behavior. These results suggest either spending less time on those topic workshops, or changing the

questions to better reflect the targeted behavior we want to measure. The State Office continues to work

with counties to use their evaluation results to help refine their program delivery by holding webinars,

face-to-face trainings, and ongoing technical assistance.

Intent to Change (ITC) – Reducing Consumption of Less Healthy Foods and Beverages

Three of the ITC topics addressed reducing consumption of less healthy foods or beverages (Table 2).

As summarized in the table, over two-thirds of participants reported that they drank a sweet beverage

every day in the past week, while over half had eaten fast food in the past week and fried foods at least

twice in the past week. Of those participants who reported practicing these undesirable behaviors, after

the workshop 75% reported the intent to drink sweet beverages “less often” and 58% intend to eat fast

food “less often” in the next week, while 61% reported the intent to eat fried food “less often.” These

results indicate considerable progress could still be made in reducing the consumption of less healthy

foods and beverages and highlight for counties key topics for future educational workshops.

Table 2. Intent To Change for Behaviors Related to Reducing the Consumption of Less Healthy Foods and Beverages


During the past week, did you eat [or drink]…

N % practicing the undesirable

behavior

Within the next week, how often will you eat [or drink]…

N % reporting intent to do it “Less

Often”

a sweet beverage every day?

1,457 72% a sweet beverage? 1055 75%

fast food? 171 52% fast food? 89 58%

fried foods 2 or more times? 295 51% fried foods? 150 61%


Adult Taste Testing Tool (n=479 tastings with 5,369 participants)

This evaluation tool is used to capture adult response to food tastings in an effort to increase exposure,

willingness, and ultimately consumption of healthy foods such as vegetables. The county nutrition

educators fill out the Adult Taste Testing Tool by asking participants five questions about their taste

testing experience.

Results

Sixteen counties utilized this tool with adult participants and found the following:

27% had ever tried the target food prior to the tasting

97% actually tried the target food in the tasting

93% would be willing to try the food again

91% were willing to serve the target food at home to their families

These results demonstrate that a large majority of adults (over 90%) introduced to novel foods (only

27% ever tried previously) find them acceptable enough to try again in the future and to serve them

to their families. Successful food tastings offer a means of increasing the quantity and variety of foods

recommended on USDA’s MyPlate to the CalFresh population. Sharing recipes featuring the target food

Page 23 of 61

provides information and skills required to incorporate the food into the household. In each county, UC

CalFresh strives to use vegetables and fruits grown locally in taste testing and to encourage planting a

garden with region/climate appropriate fruits and vegetables. Increasingly in FFY 2018, county programs

will also incorporate the promotion of local farmers’ markets that accept CalFresh EBT and Market

Match. These aspects of the UC CalFresh program help to create important linkages within the

community and environmental spheres of the Socio-Ecological Model (SEM).

Visually Enhanced Food Behavior Checklist (FBC) Evaluation (n=1080)

Background

This was the third year that county programs administered the Visually Enhanced FBC to participating

adults. The pre/post survey includes 16 questions. Of the curricula which use the FBC as the evaluation

tool, the Eat Smart Being Active curriculum is the most commonly delivered adult series. Six counties

collected surveys from a total of 1,080 adult participants. Of these participants, approximately three-

quarters (73%) self-identified as Hispanic/Latino and the great majority (82%) female.

For the pre- and post-surveys, participants are asked to report the frequency that they ate or drank a

variety of foods and beverages as well as respond to questions about their food security and general

health. Results were analyzed in two ways:

Percentage of participants showing improvement from pre- to post-survey: First, the

percentage of participants with any increase or improvement in their responses from pre to post is

reported. We defined the percent with improved behavior as the percentage of participants with

any increase in the desirable behaviors and with any decrease in the undesirable behaviors. For

example, an increase in a desirable behavior would be if a participants responded “no” to the

question “Do you eat fruits and vegetables as snacks?” for the pre-survey but for the post-survey

responded either yes - sometimes, yes- often, or yes- everyday. An example of an improvement

in an undesirable behavior would be a participant who responded “yes - everyday” to the question

“Do you drink regular soda?” in the pre-survey and then at the post-survey responded “yes-

sometimes.”

Statistically significant change from pre- to post-survey: In addition, analysis was done to

compare pre and post results for statistically significant differences.

Results

Participants making improvements in any of the desirable eating or drinking behaviors ranged

from just one in ten (8% drink milk or use milk on cereal in the past week) to nearly two-thirds (65%

improvement in cups of fruits and vegetables eaten in a day) of participants. The percentage of

participants who reported improved desirable eating behaviors are presented below in declining order:

65% - Increase in daily fruit + vegetables eaten (in cups)

56% - Increase in daily vegetables eaten (in cups)

55% - Increase in daily fruit eaten (in cups)

44% - Eat more than one kind of vegetable each day

43% - Eat more than one kind of fruit each day

41% - Eat fruits or vegetables as snacks

41% - Eat 2 or more vegetables at main meal

32% - Take skin off chicken

30% - Drink milk

21% - Have fish (in past week)

15% - Eat citrus or drink citrus juice (past week)

8% - Drink milk or use milk on cereal (past week)

Page 24 of 61

Both SMART objectives in this area were exceeded (e.g., more than 40% of participants increased their

reported frequency for eating more than one kind of vegetable and more than one kind of fruit each day).

The percentage of participants who reported improvement by reporting less frequently practicing or

experiencing the undesirable behaviors were:

23% - Ran out of food before the end of the month less frequently

28% - Drank regular soda less frequently

29% - Drank fruit drinks, sport drinks or punch less frequently

The SMART objective that 20% or greater participants would report improved food security was also

achieved.

The FBC also includes a food resource management behavior question which showed:

52% of participants reporting improvement in using the nutrition facts labels when they shop.

Again, the SMART objective of at least 50% of participants showing improvement was met.

Youth Program – FFY 2017 Evaluation Results

The majority of UC CalFresh nutrition education is provided to preschool and school-age children. Two

evaluation tools are the most commonly used across multiple direct education curricula.

Teacher Observation Tool (TOT): This tool was developed to create a retrospective evaluation

measure that could be used with the various curriculum delivered by UC CalFresh youth

programs. Teachers, youth program leaders, and other extenders at the participating sites are

trained to deliver UCCE CalFresh nutrition curricula such as Happy Healthy Me, Eating Healthy

from Farm to Fork, My Amazing Body, Good for Me and You, and It’s My Choice. The TOT

collects information on teachers’ perceptions and observations related to the changes in

knowledge and behavior among students as well as changes in their own nutrition related

practices after delivering UC developed nutrition curricula.

Teacher Taste Test Tool (TTT): In collaboration with the Evaluation Taskforce members, UC

CalFresh has developed and validated a simple TTT to evaluate youth response to food tastings

that are coupled with classroom nutrition education. These findings are included in a paper that

has been published in the Journal of the Academy of Nutrition and Dietetics1.

In addition, several curricula such as EatFit have curricula specific evaluation tools. However, in FFY

2017, these were only used by one or two counties and so they are not reported here.

Teacher Observation Tool (TOT) Results (n=1,164 classes with 28,094 students)

In 2017, 1,164 teachers completed the retrospective TOT questions on behalf of their 28,094 students

across 20 counties. Forty-three percent of these students were 1st-3rd graders, 19% were 4th-6th graders,

and the remaining 38% were preschool or kindergarten students. Over half (57%) of these TOTs were

collected in urban settings; 28% in rural settings, and the remaining 15% in suburban settings.

As a result of UC CalFresh nutrition education, the following percentage of teachers “Strongly Agree” or

“Agree” that more students now:

1 Kaiser LL, Schneider C, Mendoza C, George G, Neelon M, Roche B, Ginsburg D. Development and Use of an Evaluation Tool for Taste Testing Activities by School-Aged Children, J Acad Nutr Diet 2012; 112:2028-2034

Page 25 of 61

98% - Are able to identify healthy food choices

94% - Are willing to try new foods offered at school

83% - Wash hands more often

72% - Choose fruits and/or vegetables in the cafeteria or during classroom parties

69% - Bring fruit as a snack

Furthermore, compared to the beginning of the school year teachers also reported changes in their

own behaviors. Some highlights include teachers who report “A lot more often” engaging in the

following:

57% - Encourage students to be physically active

51% - Encourage students to eat breakfast

43% - Make healthier personal food choices

40% - Remind families to bring healthy snacks for school parties

34% - Offer healthy food choices to students (at parties, snacks, rewards)

Many of these positive changes in teacher and child behaviors move beyond the individual factors of the

Socio-Ecological Model (SEM) and are affecting environmental settings. Additionally, these findings are

supported by county reports which highlight changes in access to structured PA and classroom PA

breaks, closer collaboration between classroom and cafeteria through coordinated tastings and SLM

efforts as well as UC CalFresh staff participation on school and district School Wellness Committees.

Taste Testing Tool (TTT) Results (n=5,248 tastings with 114,320 duplicate students)

Exposure to healthy foods is particularly important for children in low-income households where

availability of fruits and vegetables is low, and limited resources discourage parents from experimenting

with new foods that their children might reject. A goal of the UC CalFresh youth program is to increase

willingness to try new healthy foods and encourage children to ask for these foods at home. Other

studies have shown willingness to try fruits and vegetables and children asking parents to buy these

foods are associated with greater household purchases of fruits and vegetables2 and fruit and vegetable

consumption in school-aged children3.

Overall, 5,248 tastings were conducted with 114,320 students from 30 counties in FFY 2017. These

numbers are compiled from multiple tastings in the classrooms sometimes with the same students.

Across all categories of healthy food items tasted in the UC CalFresh youth program:

42% of youth reported having tasted the target food before,

91% actually tried the food featured for the tasting,

64% reported willingness to eat the food again at school, and

60% reported being willing to ask for the food at home.

The results are promising in determining the students’ willingness to try the targeted foods and their

willingness to ask for this food at home. County programs can use their TTT results to make informed

choices about which foods and food groups to target next year. Findings can also be used to increase

the variety of food preferences by pairing foods less desired or novel with those generally considered as

highly appealing for food tastings to reinforce the nutrition education messages delivered.

2 Busick DB, Brooks J, Pernecky S, Dawson R, Petzoldt J. Parent food purchases as a measure of exposure and preschool-aged children’s willingness to identify and taste fruit and vegetables. Appetite 2008; 51(3): 468-473. 3 Sandeno C, Wolf G, Drake T, Reicks M. Behavioral strategies to increase fruit and vegetable intake by fourth- through sixth-grade students. J Am Diet Assoc. 2000;100(7): 828-830

Page 26 of 61

One of the SMART objectives for this evaluation tool is to have less than 40% of students report ever

trying the target food before. The intent is to expose novel foods to students (such as fruits and

vegetables that are not commonly eaten). This objective was not achieved in FFY 2017. Counties have

expressed concerns about meeting the 40% threshold established for the SMART objective when they

intentionally expose students to the same target on multiple occasions using different forms (fresh

avocado, guacamole, avocado and lime salad, etc.), as the literature shows that it often takes multiple

exposures to increase the appeal of new foods among children. To address this, the State Office worked

with county advisors to update the TTT for FFY 2018 adding questions to help capture multiple

exposures to target foods using different recipes or forms. The FFY 2018 TTT findings will be used to

reevaluate the current UC CalFresh SMART objectives for the TTT.

California Statewide Goal and Objectives

Goal 3: Improve Food Resource Management

Annually at least 80% percent of surveyed SNAP-Ed adult participants report improving at least

one food resource management behavior (such as reading labels, shopping with a list and

comparing prices to maximize use of limited resources to support a healthy diet).

Annually at least 30% of surveyed SNAP-Ed adult participants report increased food security

(defined as not running out of food at the end of the month.

Adult Resource Management – FFY 2017 Evaluation Results

Food resource management (FRM) education is one of the most requested educational trainings that UC

CalFresh offers eligible clients. The ability to successfully procure healthy foods throughout the month

while reducing instances of food insecurity (running out of food by the month’s end) can be positively

influence by an individual’s ability to assess nutritional values of available food resources, their ability to

budget their limited food dollars and their ability to critically assess the impact of food marketing on their

buying behaviors. UC CalFresh includes food resource management in two evidence based curricula:

Making Every Dollar Count and Plan, Shop, Save, Cook. Three evaluation tools are used by county

programs to assess food resource management behaviors.

Intent to Change (ITC) – Increasing Food Resource Management Practices

In FFY 2017, the county programs began using four new sets of ITC questions (make a list, plan meals,

compare unit prices, and use “Nutrition Facts”) that align with the SNAP-Ed Evaluation Framework

indicators for food resource management (ST2).

Summary results from these ITCs (Table 3) show that from one-third to one half of participants did not

currently practice these food resource management behaviors. Out of the four ITCs, using the “Nutrition

Facts” when shopping had the highest percentage (51%) of participants not practicing the desired

behavior. Of those participants not already practicing the desired food resource management behaviors,

nearly three-quarters reported the intent to make a list and over half intend to plan meals before going to

the next time they buy food, 66% reported the intent to compare unit prices before choosing foods and

62% use the “Nutrition Facts” the next time they shop. These results indicate considerable need for food

resource management skills among SNAP-Ed participants and provide an opportunity for counties to

recruit participants for series based education on these key topics.

Page 27 of 61

Table 3. Intent To Change for Behaviors Related to Increasing Food Resource Management Practices


The last time you… N % not practicing the

desirable behavior

The next time you… n % reporting intent to do

it “Yes”

Bought food, did you make a list before going to the store?

440 44% Buy food, will you make a list before going to the store?

193 72%

Bought food, did you plan meals before going to the store?

202 33% Buy food, will you plan meals before going to the store?

66 52%

Shopped, did you compare unit prices before choosing foods?

283 35% Shop, will you compare unit prices before choose foods?

98 66%

Shopped, did you use the “Nutrition Facts” on the food

label to choose foods?

1,592 51% Shop, will you use the “Nutrition Facts” on the food label to

choose foods?

815 62%


Making Every Dollar Count (MEDC) Evaluation (n=1,146)

Background

For this retrospective survey, participants are asked to rate their knowledge and skills on a scale of 1

(Low) to 5 (High) on concepts related to setting goals, resource management, meal preparation, and

food advertising both BEFORE and AFTER the program. We defined the percent with improved

knowledge as the percentage of participants with any increase or improvement on the scale from pre- to

post-program. For example, a participant could indicate BEFORE the program “Knowing simple healthy

meals to make” is “1-Low” and then a “2” AFTER the program, and that participant would count as an

individual with an increase in knowledge. In reality, participants do not report having such small

incremental improvements. The distribution of participants’ ratings BEFORE (Pre-survey) and AFTER

(Post-survey) is provided in the large table displayed on the next page.

Results

A total of 1,146 adults provided retrospective survey responses representing MEDC participants in eight

counties. Forty-four percent of participants attended classes in Spanish, while the remaining adults

attended classes in English. A majority (79%) of attendees identified as female and over half (53%)

participants reported that they or their family received CalFresh benefits. Three out of five (59%)

participants completed the eight-lesson MEDC series.

Overall, approximately three-quarters (73%-84%) of participants made improvements in

knowledge and skills for each of the MEDC measures. The three FFY 2017 SMART objectives for

MEDC were provided as a reference for gauging program performance. These are defined as a specific

minimum percentage of participants demonstrating an increase on: knowing easy ways to save money

on food (>50%); knowing simple healthy meals to make (>50%); and understanding food ads (>50%).

Results for MEDC exceeded all three SMART objectives where 79%, 74%, and 78% of participants

demonstrated improvements in knowledge respectively (Table 4).

In addition to gaining knowledge, the majority of participants reported that because of the MEDC

program they gained skills to change their behaviors (Table 5). These include setting personal goals,

using the choice-making steps with a decision they need to make, identifying community resources they

can use if needed, checking to see if they are eligible for Earned Income Tax Credit, using one of the

‘easy ways’ to save on food, and determining if using a coupon is better than buying the store brand. In

addition, nearly half (47%) of participants reported saving money, and over three-quarters (76%) were

Page 28 of 61

able to make their food last until they had money to buy more. These findings capture critical positive

outcomes in resource management among MEDC participants.

Lastly, most participants who responded to the evaluations provided feedback about the program by

answering the question “How much has the MEDC program been worth to you?” A vast majority (90%)

rated it 4 or 5 (on a 5-point scale with 5 representing the highest score), indicating it was a valuable

program for participants.

Page 29 of 61

Table 4. Comparison of Pre- and Post-Survey Responses for Making Every Dollar Count (n=1146)

Knowledge

Setting personal

goals Understanding

values

Knowing difference between need & want

How to make

choices

Knowing personal skills &

resources

Knowing community resources

Using resources to make

money go further

Knowing easy

ways to save

money on food

Knowing simple healthy meals to

make Understanding

food ads

Pre-Survey

1 Low

2

3

4

5 High

10%

28%

32%

20%

10%

8%

23%

31%

24%

14%

8%

21%

29%

22%

21%

7%

20%

32%

26%

15%

8%

25%

31%

22%

13%

13%

27%

29%

18%

13%

10%

26%

31%

20%

12%

7%

23%

32%

23%

15%

6%

21%

30%

24%

20%

12%

24%

30%

18%

16%

Post-Survey

1 Low

2

3

4

5 High

0%

1%

7%

32%

59%

0%

1%

7%

29%

62%

0%

1%

4%

23%

71%

0%

0%

6%

27%

66%

0%

1%

7%

31%

60%

0%

1%

9%

31%

59%

0%

1%

8%

28%

63%

0%

1%

6%

24%

68%

0%

1%

6%

24%

70%

0%

1%

8%

25%

66%

% Improved Knowledge

84% 80% 73% 78% 79% 80% 81% 79% 74% 78%

Smart Objective

NA NA NA NA NA NA NA > 50% > 50% > 50%

Table 5. Reported Behaviors as a Result of Making Every Dollar Count (MEDC) (n=1146)

Because of the MEDC program have you: Yes No Plan To

Written a personal goal? 70% 5% 25%

Used the choice-making steps with a decision you needed to make?

75% 5% 20%

Identified community resources you can use if needed? 78% 4% 18%

Checked to see if you are eligible for Earned Income Tax Credit? 44% 25% 31%

Used one of the easy ways to save on food? 82% 3% 15%

Determined if using a coupon is better than buying the store brand?

73% 7% 20%

Saved money? 47% 11% 42%

Made your food last until you have money to buy more? 76% 6% 18%

Page 30 of 61

Plan, Shop, Save, Cook (PSSC) Evaluation (n=1,549)

Background

This was the sixth year for the statewide use and collection of data for the UC CalFresh signature adult

curriculum: Plan, Shop, Save, Cook (PSSC). This curriculum consists of four lessons. As previously

described, PSSC was adapted based on program evaluations and participant comments so that the core

lessons could be delivered in a four-lesson series.

Evaluation of PSSC consists of a 7-item food behavior pre- and post-test. Thirteen counties collected

surveys from a subset of 1,549 participants who attended the PSSC series. Of these participants, the

majority identified as female (88%) and reported an ethnic background of Hispanic or Latino descent

(78%).

For the pre- and post-surveys, participants are asked to rate the frequency on a scale of 1 (Never) to 5

(Almost Always) in which they engage in food behaviors related to resource management and meal

planning. We defined the percent with improved behavior as the percentage of participants with any

increase or improvement on the scale of 1 to 5 from the pre- to post-survey. For example, a participant

could indicate “How often do you run out of food before the end of the month?” at pre-survey is “4-Most

of the time” and then at post-survey indicate “3-Sometimes”, and that participant would be counted as a

participant with an improvement. The distribution of participants’ ratings for both the pre- and post-survey

are provided in the table below.

Results

Participants reporting improvements in the seven PSSC behaviors and food security condition

ranged from just over one-third (36% improved food security by running out of food less often) to

three out of five (61% improved frequency of using MyPlate to make food choices) of the participants.

The six PSSC FFY 2017 SMART objectives are provided as a reference for gauging program

performance (see table below). The results indicate that the FFY 2017 SMART objectives were met for

the first five objectives listed below:

At least 40% will increase their frequency of meal planning

At least 40% will increase their frequency of using a grocery list when shopping

At least 50% will increase their frequency of using the “Nutrition Facts” on the food label to

choose foods

At least 25% will increase their frequency of comparing unit food prices

At least 30% will report that when deciding what to feed their family, they think about healthy food

choices

At least 40% will report greater food security (not running out of food at the end of a month)

The statewide objective was also met that at least 80% of surveyed SNAP-Ed adult participants

will report improving at least one food resource management behavior. Of the 1,549 participant,

completing the PSSC pre/post survey 83% reported improvement in the frequency of using at least one

of the following food resource management behaviors

Meal planning

Shopping with a list

Comparing unit prices

Thinking about healthy food choices, and

Using “Nutrition Facts” labels

Page 31 of 61

In addition, the number of participants who reported “Almost always” or “Most of the time” improved from

the pre- to post-survey and was statistically significant (p<.001) for all six PSSC behaviors (Table 5):

Up by 28% for using MyPlate to plan meals

Up by 25% for using nutrition facts label

Up by 19% for planning meals

Up by 17% for shopping with list

Up by 15% for thinking about healthy food choices

Up by 15% for comparing unit prices.

In addition, there was a statistically significant improvement in food security from pre to post with the

percent of participants who reported that they “never” or “seldom” ran out of food before the end of the

month increasing from 43% to 54% (p<.001). Although the SMART objective (≥ 40%) was not met for

“not running out of food at the end of a month,” a significant improvement in food security was reported

by PSSC participants, suggesting that the current SMART objective should be lowered to represent a

more meaningful threshold. Based on reviewing the PSSC results over multiple years, consulting other

SMART objectives related to this measure (FBC and MEDC use ≥ 20%), and the fact that food security is

impacted by a multitude of factors beyond food resource management behaviors, the threshold for this

PSSC SMART objective will be updated in FFY 2018 from “at least 40%” to “at least 30%” will report

improvement in food security as measured by not running out of food as often at the end of the month.

The percent of participants reporting “almost always” or “most of the time” for all five key PSSC

behaviors (i.e. plan, prices, shop, think, facts) increased from only one in ten (10%) at pre to over one-

third (38%) of adults at post (p<.001). These findings demonstrate both the significant gains in food

resource management behaviors as well as the positive impact on food security among PSSC

participants.


As previously described, the direct education evaluation results are used to assess progress toward

program and curricula specific SMART objectives as well as progress toward SNAP-Ed Evaluation

Framework indicators. Statewide and County-specific results are shared with UC CalFresh Program

Managers and Advisors. Program managers are encouraged to review the results with their team to

identify findings that are noteworthy as well as areas for potential improvement. Some county programs

also share evaluation results with their partners and/or sites. Originally, these surveys were intended to

capture the influence of the direct education; however, as UC CalFresh programs become more

comprehensive, they are also likely capturing to some degree the combined effects of direct education

together with PSE changes.

Point of Contact:

Questions regarding this attachment can be directed to UC CalFresh Nutrition Education Program State

Office Contacts:

Barbara MkNelly [email protected] Evaluation Analyst

Angie Keihner [email protected] Evaluation Analyst

Page 32 of 61



Table 6. Comparison of Pre- and Post-Survey Responses for Plan, Shop, Save, Cook (n=1,549)

Behavior Plan

meals

Compare unit prices

Shop with list Think about healthy choices

Use “Nutrition Facts” label

Use MyPlateα

Run out of food

Pre-Survey

Almost Always

Most of time

Sometimes

Seldom

Never

19%

26%

37%

12%

7%

22%

25%

28%

13%

12%

22%

22%

26%

14%

16%

21%

40%

29%

7%

3%

11%

20%

32%

20%

17%

6%

11%

25%

16%

41%

8%

14%

35%

22%

21%

% Always / Most 44% 47% 44% 61% 31% 17% 43% Never/Seldom

Post-Survey

Almost Always

Most of time

Sometimes

Seldom

Never

28%

35%

29%

5%

3%

29%

33%

26%

8%

5%

31%

30%

24%

9%

6%

34%

42%

19%

3%

2%

22%

34%

31%

9%

4%

16%

30%

32%

12%

10%

5%

11%

29%

30%

24%

% Always / Most 64% 61% 61% 76% 56% 45% 54% Never/Seldom

% Decreased Behavior

% Unchanged

% Improved Behavior

16%

40%

44%

21%

34%

45%

16%

41%

43%

15%

46%

39%

13%

34%

53%

9%

31%

61%

20%

44%

36%

Smart Objective > 40% will increase

frequency

> 25% will increase

frequency

> 40% will increase

frequency

> 30% will increase

frequency

> 50% will increase

frequency

NA > 40% will report greater food security

α The sample size for this question was only n=1,503 since an earlier version of the survey was used in 46 cases

Page 33 of 61


Process, Outcome and Impact Evaluation: Shaping Healthy Choices Program (SHCP) Evaluation Report FFY 2017

Submitted by Center for Nutrition in Schools – UC Davis

Project: SHCP Pilot Counties – Butte, Placer, Riverside San Joaquin, Santa Barbara, Sutter, Tulare, Yuba

The Shaping Healthy Choices Program (SHCP) is a multi-component, evidence-based, school

intervention that is based upon the Social Cognitive Theory and the Social Ecological Model to improve

children’s health and nutrition-related behaviors with a long-term goal of reducing childhood obesity. The

SHCP pilot was undertaken by UC CalFresh to better address the need for a multi-level comprehensive

delivery of an evidence-based nutrition and health program, and to address the policies, systems, and

environmental components of the school community. UC CalFresh initially piloted the program for

efficacy during the 2014-2015 academic year in four schools within three counties (Butte, Placer, and

Santa Barbara Counties). During FFY 2016 UC CalFresh continued to pilot the SHCP and expanded into

five additional counties (San Joaquin, El Dorado, Calaveras, Sutter, and Yuba). The SHCP was

implemented in FFY 2017 in two additional counties (Tulare and Riverside Counties) and discontinued in

El Dorado and Calaveras Counties; the SHCP is projected to expand into more counties in FFY 2018

and 2019.

In FFY 2017, students from 36 classrooms (n = ~1200) received nutrition education either from UC

CalFresh Educators or trained teacher extenders with support from UC CalFresh Educators (Table 1).

Table 1: Schools, classrooms and assessment by county.

Schools Classrooms Assessments

Butte County Cluster

6 15

7 classrooms: Aggregate nutrition knowledge, fidelity

8 classrooms: HCIM evaluation (data under analysis)

Placer County 1 1 Aggregate nutrition knowledge

Riverside County

2 2 Aggregate nutrition knowledge, fidelity

San Joaquin County

1 3 Aggregate nutrition knowledge, fidelity

Santa Barbara County

5 14

8 classrooms - Individual nutrition knowledge, anthropometrics, vegetable identification

6 classrooms – no assessments

Tulare County 1 1 Aggregate nutrition knowledge, fidelity

Total 16 36

These numbers include students that received nutrition education as part of a separate project at three

schools in which the physical activity curriculum, Healthy Choices in Motion (HCIM), was implemented in

addition to Discovering Healthy Choices (DHC) and Cooking Up Healthy Choices (CUHC). These

classrooms participated in the assessment of a nutrition and physical activity messaging project. Two

Page 34 of 61

classrooms were assigned to receive no messaging, three classrooms received positively-framed

messaging, and three classrooms received negatively-framed messaging. All participating students were

also provided with small, wearable accelerometers to assess minutes of physical activity. These data are

currently under analysis and are not included below.

Process Evaluation: Fidelity of Nutrition Education Delivery

During the FFY 2017 implementation of the SHCP, fidelity observations were collected on educators

responsible for facilitating lessons in the classroom. These observations were executed by Advisors,

Supervisors, Program Managers, and classroom teachers. Fidelity observations are an integral process

to ensure proper curriculum implementation in the classroom. Observations provide educators with

feedback and tools to help better align implementation with curriculum objectives and procedures. Total

fidelity is the sum of the four lesson activity sections that are each assigned two possible points,

contributing to a total possible score of eight. The four sections are:

1. Opening Questions

2. Procedure (Experiencing)

3. Sharing, Processing, and Generalizing

4. Concept and Term Discovery/Introduction

Full fidelity is achieved when the following occurs: the lesson is fully delivered, youth are interested and

engaged in the lesson, youth are attentive and actively participate in the discussion, youth are engaged

in peer-to-peer discussion for more than 75% of the lesson, and the lesson concepts are discovered by

the youth (Table 2).

Table 2: Average fidelity of implementation of the SHCP Nutrition Education Component by County

Range Combined (number of

observations)

Butte (number of

observations)

Riverside (number of

observations)

San Joaquin (number of

observations)

Tulare (number of observations)

Total fidelitya 1-8 5.43 (58) 4.88 (34) 6.11 (9) 6.36 (11) 6.00 (4)

Opening Questions

0-2 1.76 (54) 1.65 (31) 1.75 (8) 2.00 (11) 2.00 (4)

Procedure (Experiencing)

1-2 1.82 (50) 1.86 (28) 1.67 (9) 1.78 (9) 2.00 (4)

Sharing, Processing, & Generalizing

0-2 1.73 (44) 1.73 (22) 1.56 (9) 1.80 (10) 2.00 (3)

Concept, Term Discovery/ Introduction

0-2 1.43 (37) 1.39 (18) 1.50 (8) 1.40 (10) 2.00 (1)

Evidence of open-ended questionsb

1-4 3.20 (45) 3.27 (22) 2.44 (9) 3.55 (11) 3.00 (3)

Youth Engagement Ic 1-3 4.18 (45) 4.23 (22) 4.00 (9) 4.09 (11) 4.67 (3)

Youth Engagement IId 3 3.00 (43) 3.00 (21) 3.00 (8) 3.00 (11) 3.00 (3)

Youth Participatione 1-3 2.40 (43) 2.43 (21) 1.89 (9) 2.73 (11) 2.50 (2)

Concept Discovery/ Introductionf

2-3 2.66 (44) 2.67 (21) 2.44 (9) 2.73 (11) 3.00 (3)

Page 35 of 61

Range Combined (number of

observations)

Butte (number of

observations)

Riverside (number of

observations)

San Joaquin (number of

observations)

Tulare (number of observations)

Notes: aTotal fidelity is the sum of the four activity sections with a total possible score of 8: opening questions; procedure (experiencing); sharing, processing, and generalizing; and concept and term introduction/discovery. Each category: 0 = Did not do; 1 = Partially delivered according to the curriculum; 2 = Fully delivered according to the curriculum

Observations of use of inquiry-based education strategies: bOpen-ended questioning: 1 = Between 0 and 24%; 2 = Between 25 and 49%; 3 = Between 50 and 74%; 4 = Between 75 and 100%. cYouth are interested and engaged: 1 = None; 2 = about 25%; 2 = about 50%; 4 = about 75%; 5 = 100% dOverall, youth look: 1 = bored and/or preoccupied; 2 = attentive but silent; 3 = attentive and engaged in discussion eYouth participation compared to leader participation, youth talked: 1 = less than 25% of the time; 2 = about 50% of the time; 3 = more than 75% of the time fConcepts were discovered/introduced during the sessions: 1 = No concepts introduced/discovered; 2 = Concepts were not discovered and were partially introduced by educator; 3 = Concepts were discovered and/or fully discussed by youth or educator

Lesson observation sheets were to be collected for each educator 8 times (one observation per one

lesson in each module). However, one county (Placer) did not report fidelity observations and only one

county (Butte County Cluster) completed observations for each module. Even though observations were

reported for all eight modules in this county cluster, reported data was incomplete for a majority of

modules. The incomplete reporting is reflected in the inconsistent number of observations reported in the

above table. Due to incomplete data reporting, it is unclear if overall total fidelity to the curriculum

increased in FFY 2017 compared to earlier years.

Going forward, it is recommended that fidelity observations continue to be administered to ensure

implementation in the classroom matches the curriculum. The importance of collecting complete data will

be emphasized to enable analysis of implementation of all modules. One suggestion that the CNS team

will make for future years of implementation is to have classroom teachers record fidelity. This

suggestion is in response to UC CalFresh staff-reported burden on sending additional educators to

collect fidelity data.

Outcome Evaluation: Aggregate Nutrition Knowledge

Nutrition knowledge was assessed pre and post-implementation using a 35-item questionnaire. Because

individual identifiers were not used, a paired-samples t-test could not be used to compare individual pre

and post data and an independent samples t-test was used to compare pre and post scores. Nutrition

knowledge analyses were completed using SPSS 24 (IBM Corp., Armonk, NY, 2016).

Fourteen classrooms in eight schools collected aggregate nutrition knowledge. A total of 368 students

completed a pre-assessment and 373 completed a post assessment (Table 3).

Page 36 of 61

Table 3: Aggregate nutrition knowledge sample sizes pre- and post-implementation

County n-pre n-post

Butte 183 191

Placer 21 19

Riverside 66 62

San Joaquin

80 82

Tulare 18 19

Overall 368 373

A statistically significant increase in nutrition knowledge was observed from pre-implementation (20.31 ±

3.89) to post-implementation (21.79 ± 4.51; p < 0.001). (Figure 1).

Figure 1: Pre- and post-implementation nutrition knowledge scores by county.

The implementations in Placer and Tulare Counties did not result in statistically significant increases in

nutrition knowledge. This is likely a result of incomplete delivery of the DHC curriculum. In Placer County,

only one module was implemented. In Tulare County, implementation began later in the school year and

the educator was not able to deliver all modules.

AllCounties

ButteCountyCluster

PlacerCounty

RiversideCounty

SanJoaquinCounty

TulareCounty

Pre 20.3 21.17 21.81 18.91 19.33 19.32

Post 21.79 22.19 23.63 21.13 21.46 19.39

0

5

10

15

20

25

30

35

To

tal S

co

re o

ut

of

35

Nutrition Knowledge

* * * *

*p<0.05

*

Page 37 of 61

Impact Evaluation: Individual-level Nutrition Knowledge, Vegetable Identification, and

Anthropometrics

The purpose of this evaluation was to determine the effectiveness of utilizing an extender model for

implementation of the SHCP compared to direct education, which has been demonstrated to be effective.

In the extender model, classroom teachers provided the education with support from UC CalFresh

educators. The SHCP was implemented in three schools in Santa Barbara County. Overall, Santa

Barbara County had a consent rate of 62%, with 187 out of 304 parents/guardians consenting to their

child’s participation in the assessment of the SHCP. However, the number of students completing each

assessment varied and are reported with the results for each assessment.


The evaluation costs are integrated into the overall cost implementation of the program.

Project Goals:

This evaluation supports the following California SNAP-Ed State Level Goal:



Evaluation Design:

Evaluation participants: The SHCP was implemented in three schools in Santa Barbara County.

Students in eight classrooms received nutrition education, which included inquiry-based, garden-

enhanced nutrition education from the curriculum Discovering Healthy Choices (DHC) as well as cooking

demonstrations from Cooking up Healthy Choices (CUHC). One classroom in one school served as a

comparison classroom and received no SHCP programming.

Assignment to intervention and control or comparison conditions:

Unit of Assignment: Unit of assignment was school, with two schools (8 classrooms) assigned to the

intervention group, and one schools (1 classroom) assigned to comparison group.

Group Assignment: This assignment of the comparison group was not random and was based on the

school not having participated in the SHCP in prior years.

Outcome and Impact Measures, Data Collection, and Results:

Nutrition Knowledge

Nutrition knowledge was assessed pre and post-implementation using a 35-item questionnaire. Analysis

of Covariance (ANCOVA) was conducted to assess the changes in nutrition knowledge in the direct

education and extender classrooms compared to the control classroom. To control for pre-

implementation score it was included as a covariate in the analysis. Paired samples t-tests were also

used to compare pre- and post-implementation scores. Nutrition knowledge analyses were completed

using SPSS 24 (IBM Corp., Armonk, NY, 2016).

Nine classrooms collected nutrition knowledge. A total of 147 students completed pre- and post-

assessments (direct education n = 47, extender n = 75, comparison n = 25). A statistically significant

increase in nutrition knowledge was observed from pre-implementation to post-implementation in the

direct education (pre = 17.74 ± 4.1; post = 21.04 ± 5.4; p < 0.001), and extender (pre = 20.65 ± 3.6; post

= 23.07 ± 3.5; p < 0.001) groups, but not in the comparison group (pre = 21.32 ± 3.9; post = 22.4 ± 3.9; p

= .213) (Figure 2).

Page 38 of 61

Figure 2: Nutrition knowledge pre- and post-implementation in each group.

When an ANCOVA analysis was conducted to compare the direct education and extender groups to the

comparison, no statistically significant difference was detected (p = 0.249). This would suggest that the

direct education and extender groups did not experience an increase in nutrition knowledge compared to

the comparison. However, the comparison group had a much smaller sample size than the direct

education and extender groups, which may have limited the ability to detect differences. It is possible that

with a larger comparison group there may be detectable differences.

Anthropometrics

Height and weight were assessed pre- and post-implementation and were used to calculate body mass

index (BMI) percentile. Analysis of Covariance was conducted to assess the changes in BMI percentile in

the direct education and extender classrooms compared to the control classroom with pre-

implementation BMI as a covariate.

Nine classrooms collected anthropometric data. A total of 138 students completed a pre- and post-


decrease in BMI percentile was observed from pre-implementation to post-implementation in the direct

education group (pre = 78.78 ± 21.46; post = 73.97 ± 21.5; p = 0.01), but not in the extender (pre = 85.28

± 17.9; post = 83.85 ± 20.3; p = 0.82) or comparison groups (pre = 73.59 ± 27.9; post = 73.40 ± 27.3; p =

0. 859) (Figure 3).

ComparisonDirect

EducationExtender

Pre 21.32 17.74 20.65

Post 22.24 21.04 23.07

0

5

10

15

20

25

30

35

To

tal S

co

re o

ut

of

35

Nutrition Knowledge

* *

*p<0.05

Page 39 of 61

Figure 3: BMI Percentile by Treatment Group


comparison, a statistically significant difference was detected (p = 0.024). Post-hoc analysis indicated

that the mean difference between the direct education and comparison groups was statistically significant

(p = 0.042), indicating that the direct education group experienced a decrease in BMI percentile

compared to the comparison group.

Vegetable Identification

Ability to identify ten different vegetables was assessed pre- and post-implementation. In addition to

vegetable identification, students were also asked if they consumed the food at home, if they would ask

their family to purchase this food, and if they would eat this food as a snack (data not shown). Analysis of

Covariance was conducted to assess the changes in vegetable identification in the direct education and

extender classrooms compared to the control classroom with pre-implementation identification included

as a covariate.

Nine classrooms collected vegetable identification data. A total of 136 students completed pre- and post-


increase in overall vegetable identification was observed from pre-implementation to post-implementation

in the extender group (pre = 3.72 ± 1.38; post = 5.03 ± 1.39; p < 0.001), but not in the direct education

group (pre = 3.68 ± 1.08; post = 3.44 ± 1.69; p = 0.279) or comparison groups (pre = 3.52 ± 1.24; post =

4.48 ± 1.68; p = 0.059) (Figure 4).

Comparison Direct Education Extender

Pre 73.59 78.78 85.28

Post 73.40 73.97 83.85

0

20

40

60

80

100

BM

I P

erc

en

tile

BMI Percentile

*

*p<0.05

Page 40 of 61

Figure 4: Vegetable Identification by Treatment Group


comparison, a statistically significant difference was detected (p < 0.000). However, post-hoc analysis

indicated that the mean difference between the direct education and comparison groups was statistically

significant (p = 0.026), suggesting that the direct education group experienced a decrease in vegetable

identification compared to the comparison group.

Analyses were conducted for each individual vegetable, however only snap peas in the extender group

(13% identified) demonstrated significant improvements compared to the comparison group (0%

identified) when controlling for pre-assessment score (pre score = 6%, p = 0.004).


The evaluation results will be used to continue to make research-based improvements to the SHCP.

These results demonstrate that the extender model for education is feasible and has similar results to

direct education. The extender model will continue to be expanded to other counties that participate in

the SHCP.

Impact Evaluation: Healthy Choices in Motion

To evaluate if the Healthy Choices in Motion curriculum improves physical activity knowledge, a 20-item

questionnaire was developed using the primary learning concepts. Experts reviewed the questionnaire

for content validity and a test and a retest procedure was used to determine reliability and internal

validity.


Evaluation costs were covered through a non SNAP-Ed-funded grant.

Project Goals:

This evaluation supports the following California SNAP-Ed State Level Goal:

Goal 2: Increase Physical Activity

Comparison Direct Education Extender

Pre 3.52 3.69 3.72

Post 4.48 3.44 5.03

0

1

2

3

4

5

6

7

8

9

10

To

tal S

co

re o

ut

of

10

Vegetable Identification

*

*p<0.05

Page 41 of 61

Evaluation Design:

Evaluation participants: The HCIM lessons were evaluated in a five-week, quasi-experimental pilot

study. Eight classes participating in the UC CalFresh Nutrition Education Program were invited to

participate. Six classes were selected because of their current participation in the SHCP. Two of the

SHCP-classes received the HCIM lessons and two other SHCP-classes received the HCIM lessons with

an added technology component. The added technology component consisted of clip-on accelerometers

to assess physical activity that were worn by the students.

Assignment to intervention and control or comparison conditions:

Unit of Assignment: Unit of assignment was school, with two schools (6 classrooms) assigned to the

intervention group, and one school (two classrooms) assigned to comparison group.

Group Assignment: This assignment of the comparison group was not random and two classrooms in

one school were selected to be the comparison group due to not receiving direct nutrition education.

Outcome and Impact Measures, Data Collection, and Results:

The objective was to improve physical activity knowledge in 4th and 5th grade students. At pre- and post-

intervention assessments, students in the comparison group (n=23), SHCP group (n=19), SHCP + HCIM

group (n=36), and SHCP + technology-enhanced HCIM group (n=40) completed a 20-item physical

activity knowledge questionnaire. Total physical activity knowledge scores were calculated into means

and standard deviations for each group and distributions were examined for normality using a

combination of histogram, skewness, and Kurtosis. Change in physical activity knowledge from pre- and

post- was calculated by subtracting post-scores from pre-scores. Analyses were conducted using STATA

14.0. Tests performed were ANOVA and Bonferroni for multiple comparisons. Students that participated

in HCIM demonstrated significant improvements in physical activity knowledge compared to the

comparison group (2.8, P=0.01) and SHCP group (3.0, P=0.01). Students that participated with the

technology-enhancement also demonstrated improvements in physical activity knowledge compared to

the SHCP group (2.3, P=0.05). There were no differences when comparing the technology-enhancement

to the comparison group (2.1, P=0.08) and HCIM group (-0.7, P=1.0). Improvements in physical activity

knowledge in students participating in HCIM with and without the technology-enhancement may

contribute to improvements in physical activity-related behaviors and should be further explored over a

longer intervention period.


The evaluation results will be used to make research-based improvements to the HCIM curriculum. The

results are also currently being prepared for submission to a peer-reviewed journal.

Outcome Evaluation: Shaping Healthy Choices School Health Check (SHC²)

The Shaping Healthy Choices Program has created a tool, known as the Shaping Healthy Choices

School Health Check (SHC²), which allows school site stakeholders to evaluate how well their site is

complying with school wellness policy requirements. The tool is formatted to resemble a rubric, with

scores ranging from “Not in Place” and “Meets Standard” to “Exceeds Criteria.” The tool includes

questions that align with the four components of the Shaping Healthy Choices Program, Nutrition

Education and Physical Activity Promotion, Family and Community Partnerships, Foods Available on

Campus with an emphasis on Local and Regional Agriculture, and School-site Wellness. Recently, the

tool has been moved from a paper form to an electronic spreadsheet to make it easier for stakeholders to

identify areas that they are doing well in and areas that could be improved. The tool also includes a

summary sheet that highlights specific questions that were marked lower than the standard, as well as a

goal setting sheet designed to promote partnerships in achieving successful site changes.

Page 42 of 61

Compliance with wellness policy requirements was assessed for each school pre- and post-

implementation and a paired-samples t-test was conducted to compare responses. Only schools in which

there were both pre- and post-data were included in the analyses. Statistical analyses were conducted

using SPSS 24 (IBM Corp., Armonk, NY, 2016).

Overall, there was no statistically significant increase from pre- (51.68 ± 13.90) to post-implementation

(56.57 ± 12.86) in total SHC2 percent (p = 0.056) (Table 4). A statistically significant increase (p = 0.040)

in Nutrition Education and Promotion was observed from pre- (45.67 ± 14.19) to post-implementation

(52.38 ± 10.05). Increases were observed in all other categories; however these were not statistically

significant.

While the majority of sections and overall score was not found to be significant, the observed increases

suggest that schools that participate in the SHCP are continuing to improve their adherence to wellness

policy requirements.

Table 4: Pre- and post-implementation SHC2 percent.

SHC² Section (n) FFY 2016 SHC2

Percent (SD) FFY 2017 SHC2

Percent (SD) p-value

Nutrition Education and Promotion (14)

45.67 (14.09) 52.38 (10.05) 0.040*

Family and Community Partnerships (14)

45.00 (18.39) 49.52 (17.14) 0.256

Foods Available on the School Campus (14)

72.86 (15.35) 71.67 (17.86) 0.755

Regional Agriculture (13)

52.56 (26.43) 53.21 (24.66) 0.907

Wellness (13)

55.45 (18.67) 59.94 (22.98) 0.495

Overall (14)

51.68 (13.90) 56.57 (12.86) 0.056

Outcome Evaluation: Impact of SHCP on Academic Metrics

There has been recent effort to collect school-level data to extrapolate information about the large-scale

effects of the Shaping Healthy Choices Program. Factors that affect school site funding like absenteeism,

school meal participation, and standardized test scores will be evaluated to determine the potential

effects that the Shaping Healthy Choices Program has on overall school funding.

Project goals and objectives were briefly presented to educators in March of 2017. Educators were

asked to contact school administrators for schools participating in the Shaping Healthy Choices Program

as well as one school in the same county. Between April and June, one school administrator and one

school nutrition services member from each educator-contacted school were approached to provide

additional details about the project as well as a full explanation of the requested information. There are

some schools that were directly contacted by UC CalFresh as well.

The requested data for each school included attendance data for each student, organized by days

enrolled and days absent per month and average daily participation for each class, organized by the type

of meal (full priced, reduced price and free meals) by month.

Page 43 of 61

Schools that were contacted include schools in following county clusters: Butte, Placer/Nevada, San

Joaquin, Santa Barbara/San Luis Obispo, Riverside, and Tulare.

Ultimately, the goal with these data is to be able to determine if there are connections between SHCP

involvement and attendance, test scores, and school meal participation. Optimally, the data would be

able to be tracked by student without having identifiers present. However, as the data were returned,

there were challenges tracking on a student-by-student level basis.

It is evident that innovative strategies are needed to properly collect the data from the schools without

high administrator burden. Schools use different reporting software programs, making it difficult to make

the data uniform for meaningful analysis. Data that have already been collected will be evaluated to

determine methods for improving data collection in the upcoming year. Meetings will also be arranged

with select food service directors and school administrators to optimize the protocol for data collection.

Process Evaluation: Educator Self-efficacy

The objective of this evaluation was to assess educator self-efficacy beliefs after participating in the

SHCP professional development program that utilized an action-inquiry approach.

Data were collected using a retrospective questionnaire to avoid response shift bias. A convenience

sample of UC CalFresh nutrition educators (n=10) participating in the SHCP reported on their self-

efficacy beliefs after participating in a 10-month professional development program designed to help

educators understand and adopt an inquiry-based approach to learning and teaching nutrition and health

related concepts. A paired-samples t-test was conducted to compare responses from the retrospective,

self-efficacy questionnaire. Statistical analyses were conducted using SPSS 24 (IBM Corp., Armonk, NY,

2016).

Many of the variables in this study exhibited statistically significant increases in measurements from pre-

to-post and indicated that participants experienced growth in teaching efficacies, increased confidence to

teach nutrition, and improved facilitation skills for guiding inquiry in the classroom (Table 5).

Table 5: Mean Scores and Standard Deviations for Educator Self-Efficacy Before and After Participation in Professional Development

Questions Post Score Mean

(SD)

Pre Score Mean

(SD)

p-value

Q1A: After participating in the SHCP, I believe I can do a good job teaching students about nutrition.

Q1B: Prior to participating in the SHCP, I believed I could do a good job teaching students about nutrition.

4.50 (.707) 3.60 (1.265) .054

Q2A: After participating in the SHCP, I am able to stimulate students enough so they ask thoughtful questions about nutrition.

Q2B: Prior to participating in the SHCP, I was able to stimulate students enough so they ask thoughtful questions about nutrition.

4.40 (.699) 3.10 (.994) .002 *

Q3A: After participating in the SHCP, I believe I can do a good job teaching students about consumerism.

Q3B: Prior to participating in the SHCP, I believed I could do a good job teaching students about consumerism.

4.80 (.422) 3.90 (.738) .001 *

Page 44 of 61


(SD)

Pre Score Mean

(SD)

p-value

Q4A: After participating in the SHCP, I believe I can evaluate improvements in nutrition skills.

Q4B: Prior to participating in the SHCP, I believed I could evaluate improvements in nutrition skills.

4.60 (.516) 3.70 (.949) .019 *

Q5A: After participating in the SHCP, I believe I can do a good job teaching students about nutrients.

Q5B: Prior to participating in the SHCP, I believed I could do a good job teaching students about nutrients.

4.40 (.843) 3.20 (1.751) .044 *

Q6A: After participating in the SHCP, I believe increased teaching time in nutrition produces significant changes in nutrition-related behaviors of many students.

Q6B: Prior to participating in the SHCP, I believed increased teaching time in nutrition produces significant changes in nutrition-related behaviors of many students.

4.60 (.516) 3.90 (.994) .066

Q7A: After participating in the SHCP, I believe the students I teach will become more knowledgeable about nutrients.

Q7B: Prior to participating in the SHCP, I believed the students I taught would become more knowledgeable about nutrients.

4.80 (.422) 3.60 (.843) .003 *

Q8A: After participating in the SHCP, I believe the students I teach will be more knowledgeable about the recommendations for a healthy diet.

Q8B: Prior to participating in the SHCP, I believed the students I taught would be more knowledgeable about the recommendations for a healthy diet.

4.70 (.675) 3.80 (.919) .019 *

Q9A: After participating in the SHCP, I act as a facilitator for youth as they work on their activities.

Q9B: Prior to participating in the SHCP, I acted as a facilitator for youth as they work on their activities.

4.40 (.966) 2.80 (1.135) .000 *

Q10A: After participating in the SHCP, I understand how to lead a group of students through the inquiry-based learning process.

Q10B: Prior to participating in the SHCP, I understood how to lead a group of students through the inquiry-based learning process.

4.70 (.483) 2.70 (1.337) .001 *

Q11A: After participating in the SHCP, I believe that using an inquiry-based approach is an effective way to learn and teach.

Q11B: Prior to participating in the SHCP, I believed that using an inquiry-based approach is an effective way to learn and teach.

4.70 (.483) 3.50 (.850) .003 *

Page 45 of 61


(SD)

Pre Score Mean

(SD)

p-value

Q12A: After participating in the SHCP, I ask youth open-ended questions, such as, "Explain what you know about XX.”

Q12B: Prior to participating in the SHCP, I asked youth open-ended questions, such as, "Explain what you know about XX.”

4.80 (.422) 3.00 (1.054) .000 *

Q13A: After participating in the SHCP, I believe that participating in a community of practice (COP) is an effective way to strengthen educator's skills.

Q13B: Prior to participating in the SHCP, I believed that participating in a community of practice (COP) is an effective way to strengthen educator's skills.

4.60 (.699) 3.80 (.919) .037 *

Q14A: After participating in the SHCP, I teach youth through direct instruction, using techniques such as lectures or demonstrations.

Q14B: Prior to participating in the SHCP, I taught youth through direct instruction, using techniques such as lectures or demonstrations.

3.30 (1.337) 3.90 (1.197) .217

Q15A: After participating in the SHCP, I encourage youth to apply concepts they learn to new situations.

Q15B: Prior to participating in the SHCP, I encouraged youth to apply concepts they learn to new situations.

4.30 (.823) 3.80 (.919) .052

Utilizing an action-inquiry approach for educator professional development is an effective way to increase

educator confidence to teach nutrition and health related topics and improve facilitation skills for guiding

inquiry in the classroom.

Point of Contact:

Questions regarding the content of this attachment can be directed to:

Anna M. Jones, PhD

Shaping Healthy Choice Program

Center for Nutrition in Schools

University of California Davis

Phone: 530-752-3387

Email: [email protected]

Page 46 of 61


1825 Bell Street, Suite 102, Sacramento, CA 95825 | T: 916.265.4042 | F: 916.265.4043 | www.CenterForWellnessAndNutrition.org


Process Evaluation: UC CalFresh Youth Engagement Initiative

Mid-Point Assessment Report

Submitted by Public Health Institute Center for Wellness and Nutrition


UC CalFresh Youth Engagement

Mid-Point Assessment Report

Kyli Gallington and Metria Munyan

Public Health Institute Center for Wellness and Nutrition

December 1, 2017

Page 47 of 61

BACKGROUND

In FFY 2016 the Public Health Institute (PHI) Center for Wellness and Nutrition (CWN) and The

University of California CalFresh Nutrition Education Program (UC CalFresh) partnered on a Youth

Engagement Pilot Project funded through SNAP-Ed. During the first project year, six UC CalFresh

pilot sites were identified and confirmed to explore and document existing, scalable, and innovative

strategies to engage young people. CWN developed and administered a survey in November 2015

statewide to capture baseline information assessing readiness and current youth engagement levels.

Based on results, CWN explored supportable ways to build capacity and foster adult/youth

partnerships with the goal of increasing youth engagement strategies and leadership opportunities

throughout UC CalFresh and across California.

In FFY 2017 UC CalFresh expanded statewide youth engagement partnerships with UC Davis

Center for Regional Changes (CRC) and California 4-H programs. CWN, CRC, and California 4-H

collaborated to provide a broad base of training and technical assistance to all UC CalFresh sites

interested in expanding youth engagement strategies. In October 2017, CWN developed and

administered a mid-point assessment survey across the state to gauge engagement levels and

progress on increasing youth engagement strategies.

SUMMARY OF KEY FINDINGS

A statewide, mid-point assessment survey was administered to the UC CalFresh County/County

Cluster programs to capture the current work taking place across the state. The survey asked about

youth engagement in Elementary School, Middle School, High School, and Afterschool settings;

Policy, Systems, and Environmental (PSE) activities; Youth-Led Participatory Action Research

(YPAR); youth leadership opportunities; key partnerships; and resources / trainings. Advisers,

Supervisors, Managers, and Educators from each of the UC CalFresh counties or county clusters

were invited to participate in the survey. Responses to the survey were compared to responses

received on the baseline assessment that was administered in 2015.

Respondents

Respondents to the survey (n=14) represented 10 of the 15 UC CalFresh Counties or County

Clusters that were invited to participate (Figure 1). For 4 of the UC CalFresh Counties/County

Clusters, responses were received from 2 representatives. In comparison with the baseline

assessment, respondents (n=15) represented 13 of the 15 Counties or County Clusters (Figure 1),

and for 2 of the Counties/County Clusters responses were received from 2 representatives.

Page 48 of 61

Figure 1. Respondents of the Baseline (n=15) and Mid-Point (n=14) Assessments by UC CalFresh County or County Cluster.

Mid-point assessment respondents varied by role, including 6 Supervisors, 5 Educators, 2 Advisers,

and 1 Manager. In contrast, baseline assessment respondents included 7 Supervisors, 4 Advisers,

and 4 Managers, but no Educators.

Alameda County

Amador / Calaveras /El Dorado / Tuolumne Counties

Butte / Colusa / Sutter / Glenn / Yuba Counties

Fresno / Madera Counties

Imperial County

Kern County

Placer / Nevada Counties

Riverside County

San Joaquin County

San Luis Obispo / Santa Barbara Counties

Santa Clara / San Francisco / San Mateo Counties

Shasta / Trinity / Tehama Counties

Stanislaus / Merced Counties

Tulare / Kings Counties

Yolo County

0 1 2 3

Frequency (n)

Respondents by UC CalFresh County or County Cluster

Mid-Point Baseline

Page 49 of 61

Elementary Schools

Almost all respondents (93%) indicated that their UC CalFresh program works with youth in upper

Elementary School, grades 4-6.

Table 1. Snapshot of work with upper Elementary School youth among respondents (n=13).

% %

Grade levels Settings where youth are engaged

4th Grade 77 During the school day 77

5th Grade 92 After the school day 69

6th Grade 77 In the classroom 54

Number of Elementary Schools In the garden 46

1 to 4 31 In the cafeteria 39

5 to 10 39 In an afterschool program 54

11 to 25 23 Other1 23

26 to 50 8 1Other responses included before school and in off-campus settings such as food bank distributions.

Table 2. Baseline and mid-point comparison of statewide with Elementary School youth by county.

Elementary Schools Baseline Assessment

Elementary Schools Mid-Point Assessment

County/ County Cluster

GRADE

LEVELS2

# OF

SCHOOLS

GRADE

LEVELS2

# OF

SCHOOLS

Alameda No Programing Reported No Response

Central Sierra Cluster K-6th 11-25 4th - 6th 1-4

Fresno/Madera MCP K-6th 26-50 4th - 6th 26-50

Imperial K-6th 26-50 6th 1-4

Kern No Programing Reported No Response

Placer/Nevada1 K-6th 5-10 4th - 5th 5-10

Riverside K- 3rd 5-10 No Programing Reported

San Joaquin K-6th 11-25 No Response

San Luis Obispo/ Santa Barbara Counties1 K-6th 5-10 5th - 6th 5-10

Santa Clara/San Mateo/SF K-6th 26-50 4th - 6th 5-10

Shasta / Trinity / Tehama Counties1 No Response 4th - 6th 11-25

Stanislaus/Merced K-6th 11-25 No Response

Tulare / Kings Counties 4th - 5th 11-25 4th - 5th 1-4

Yolo County1 K-6th 11-25 4th - 6th 11-25 1Counties with multiple responses, the highest response is presented 2The 2015 Baseline Assessment gathered elementary data for K-6th grades and the 2017 Mid-Point Assessment

gathered data relative to upper elementary 4th – 6th grades

Page 50 of 61

Youth engagement opportunities at the elementary level shifted slightly from baseline to the mid-

point assessment. Respondents indicated that:

Youth leading Elementary School activities increased from 54% to 69%

Youth initiating action in Elementary School activities increased from 31% to 39%

Youth involved in shared decision-making with adults at the Elementary School level

increased from 42% to 62%

When asked to describe how youth lead activities, initiate actions, or share in decision-making,

respondents highlighted the following:

“They have assigned roles, provide opinions and feedback, assist with peer to peer

education, etc.” – Supervisor

“Students who have completed Cooking Academy return to lead cooking groups for the

new class of students.” – Educator

“Decide on projects to address health issues in their school or community.” - Educator

“We discuss and decide with our student leaders how we should deliver the nutrition lesson

component and activities.” – Educator

Within the Elementary School setting, the most common programs, activities, curricula, or projects

being implemented were:

Structured physical activity programs such as CATCH or Spark (77%)

Smarter Lunchrooms Movement (77%)

Cooking classes or Cooking Academy (69%)

Harvest of the Month activities (69%)

Garden programs (69%)

The top challenges reported in the mid-point assessment remain similar to those reported in the

original assessment, with the addition of 54% reporting difficulty being able to dedicate staff time to

youth efforts. Otherwise, time limitations (61%), competing priorities (46%), and funding/resources

(38%) continue to be common barriers for youth engagement programing at the elementary level.

Notably, other responses included meeting places, starting this process, program capacity, and

policies that don’t support youth engagement in low-income communities (transportation, etc.).

Page 51 of 61

Middle Schools

Less than half of respondents (43%) indicated that their UC CalFresh program works with youth in

Middle School, grades 6-8. Most respondents reported engaging youth in the classroom (67%) Half

of the respondents indicated that all of the Middle Schools they work with enroll students coming

from Elementary Schools that their program also works with, and a third of respondents (33%)

indicated that their Middle School level programs interact with or teach back to UC CalFresh

Elementary School Programs. Examples of this include teens as teachers in the Cooking Academy

or Healthy Living Ambassador program.

Table 3. Snapshot of work with Middle School youth among respondents (n=6).

% %





Number of Middle Schools In the garden 33

1 to 4 50 In the cafeteria 33


11 to 25 17

26 to 50 0


Middle Schools Baseline Assessment

Middle Schools Mid-Point Assessment

County/ County Cluster

GRADE

LEVELS

# OF

SCHOOLS

GRADE

LEVELS

# OF

SCHOOLS


Central Sierra Cluster 6th - 8th 1-4 No Programing Reported

Fresno/Madera MCP 5-10 No Programing Reported

Imperial 6th - 8th 1-4 6th - 8th 1-4


Placer/Nevada1 6th - 8th 1-4 No Programing Reported

Riverside 6th - 8th 5-10 6th - 8th 5-10

San Joaquin No Programing Reported No Response

San Luis Obispo & Santa Barbara Counties1 6th - 8th 1-4 No Programing Reported

Santa Clara/San Mateo/SF 6th - 8th 1-4 6th - 8th 1-4


Stanislaus/Merced (2015 Assessment) 6th - 8th 5-10 No Response

Tulare / Kings Counties 6th - 8th 5-10 No Programing Reported

Yolo County1 6th - 8th 1-4 1-4 1Counties with multiple responses, the highest response is presented

Page 52 of 61

Youth engagement in the Middle School setting has increased from baseline to the mid-point

assessment. Respondents indicated that:

Youth leading activities increased from 10% to 67%

Youth initiating action increased from 0% to 33%

Youth involved in shared decision making with adults increased from 10% to 50%

Respondents reported that youth lead and initiate actions through YPAR projects. Shared decision-

making, as described by one respondent, included serving on wellness committees:

“In some of our schools, youth are on wellness committees.” – Educator

Within the Middle School setting, the most common programs, activities, curricula, or projects being

implemented were:


Cooking classes or Cooking Academy (40%)



Teens as Teachers (40%)

YPAR projects (40%)

Most mid-point assessment responses regarding challenges from those working with Middle School

youth were again comparable to the baseline assessment, indicating that time limitations (80%) and

competing priorities (40%) continue to be consistent barriers. Additionally, being able to dedicate

staff time to youth efforts was a new top challenge, reported by 60% of respondents.

Page 53 of 61

High Schools

Half of respondents (50%) indicated that their program works with youth in High School, grades 9-

12. Three respondents (43%) indicated that all or most of the High Schools they work with enroll

students coming from Middle Schools that their program also works with, and three respondents

(43%) indicated that their High School level programs interact with or teach back to UC CalFresh

Elementary School or Middle School Programs.

Table 5. Snapshot of work with High School youth among respondents (n=7).

% %





12th Grade 71 In the garden 0

Number of High Schools In the cafeteria 14


5 to 10 0

11 to 25 0

26 to 50 0


High School Baseline Assessment

High School Mid-Point Assessment

GRADE

LEVELS

# OF

SCHOOLS

GRADE

LEVELS

# OF

SCHOOLS


Central Sierra Cluster No Programing Reported 11th-12th 1-4

Fresno/Madera MCP No Programing Reported No Programing Reported

Imperial No Programing Reported No Programing Reported


Placer/Nevada1 9th - 12th No Programing Reported

Riverside 9th - 12th 5-10 9th - 10th 1-4

San Joaquin No Programing Reported No Response

San Luis Obispo & Santa Barbara Counties1 12th 1-4 10th - 12th 1-4

Santa Clara/San Mateo/SF 9th - 12th 5-10 9th & 11th 1-4


Stanislaus/Merced (2015 Assessment) No Programing Reported No Response

Tulare / Kings Counties 9th 1-4 No Programing Reported

Yolo County1 9th - 12th 1-4 9th - 12th 1-4 1Counties with multiple responses, the highest response is presented

Page 54 of 61

Youth engagement opportunities at the High School level shifted from baseline to the mid-point

assessment. Respondents indicated that:

Youth leading activities in High School programing decreased from 67% to 29%

Youth initiating action in High School activities decreased from 33% to 14%4

Youth involved in shared decision making with adults at the High School level increased

from 17% to 29%

When asked to describe how High School youth lead activities, respondents indicated that youth

lead through CATCH activities or as teens as teachers in the HLA program. Examples of shared

decision-making, as described by respondents, included making posters for Smarter Lunchrooms

Movement or participating in the HLA program planning.

Within the High School setting, the most common programs, activities, curricula, or projects being

implemented were:




Teens as Teachers (40%)

Most respondents (57%) of the mid-point assessment that are working with High School youth

indicated that time limitations, being able to dedicate staff time to youth efforts, and balancing

competing priorities are the top challenges they face when engaging youth. These results are

consistent with the baseline assessment. Other notable challenges included starting the process and

logistical challenges, and program capacity.

Respondents described successes they have had where youth input or involvement helped improve

the program and/or build youth skills and capacity at the High School level:

“…They were trained to teach younger youth; helped integrate P.E. into younger grade

levels who do not have P.E. specialists; some media and marketing was involved when this

program came about.” – Educator

Afterschool / Community Programs

Most respondents (71%) indicated that their UC CalFresh program works with youth in afterschool

programs and/or community-based organizations, including upper Elementary

School, Middle School, and High School Youth. Most respondents (80.0%) reported engaging youth

in Afterschool programs.

4 It should be noted that three respondents from the baseline assessment reported working in high schools, but subsequently did not report working in high schools in the midpoint assessment. Refer to Limitations for further discussion about factors that may have affected changes in responses between the baseline and mid-point assessments.

Page 55 of 61

Table 7. Snapshot of work with youth in afterschool or community-based settings among respondents (n=10).

% %


Upper Elementary 100 Afterschool program 80

Middle School 50 Summer camp programs 40

High School 20 Boys and Girls Club 30

Number of Programs/Organizations YMCA 20

1 to 4 70 Community center 20

5 to 10 30 Youth center 20

11 to 25 0 Community gardens 20

26 to 50 0 4-H Club 10

City or County program 10

There has been a positive shift in youth engagement opportunities within afterschool or community-

based organization settings from baseline to the mid-point assessment. Respondents indicated that:

Youth leading activities in afterschool and/or community-based programing increased from

20% to 30%

Youth initiating action in afterschool and/or community-based program activities increased

from 0% to 30%

Youth involved in shared decision making with adults in the afterschool and/or community-

based environment increased from 20% to 40%

When asked to describe how youth lead activities, respondents indicated that youth lead through

YPAR projects or as teens as teachers. Respondents reported that youth initiate actions such as

“Nutrition Day” or within the YPAR project. Examples of shared decision-making, as described by

respondents, included participating in YPAR, the HLA planning, or “implementing cafeteria tasting.”

Within afterschool or community-based organization settings, the most common programs, activities,

curricula, or projects being implemented were:

Cooking classes or Cooking Academy (70%),

Structured physical activity programs such as CATCH or Spark (40%),


Most respondents of the Mid-Point Assessment reported working with youth in afterschool or

community-based organization settings indicated that time limitations (80%) and being able to

dedicate staff time to youth efforts (60%) are the top challenges they face when engaging youth.

Competing priorities and retaining youth involvement were also reported by 40% of respondents.

Page 56 of 61

Half of respondents shared successes where youth input or involvement helped improve the

program and/or build youth skills and capacity in afterschool or community-based organization

settings. Respondents described successes related to collaboration and implementation of

programs:

“…They were trained to teach younger youth; helped integrate P.E. into younger. Have

developed Cooking Academy over the past couple of years and have seen youth expand

their skills. It has improved our local program delivery.” – Adviser

“YPAR project led to a great relationship and acceptance of more UC CalFresh

Programming.” – Educator

Policy System and Environmental (PSE) Change

Every respondent indicated that their UC CalFresh program is involved in at least one PSE change

that directly affects youth or youth serving organizations. When asked about specific PSE changes,

respondents indicated most often that they are involved in:


Community and School Gardens (93%)

New or improved access to structured physical activity (79%)

None of the respondents indicated that their program is currently involved in Joint Use Agreements

or Healthy Vending Standards. Reports were similar at baseline and mid-point. However, by

comparison, some PSE activities were reported by a higher percentage of respondents at mid-point

(e.g., Stencil/Mural Projects, etc.) while other PSE activities were reported by a lower percentage of

respondents at mid-point (e.g., Free Drinking Water, etc.).

Table 8. Percentage of respondents that indicated their UC CalFresh Program is involved in the following PSE changes that

directly affect youth or youth serving organizations at Baseline and Mid-Point.

Baseline Mid-Point

% %

Community and School Gardens 86% 93%

Smarter Lunchrooms Movement 93% 93%

New/Improved Access to Structured Physical Activity1 - 79%

Stencil/Mural Projects 36% 57%

Model Local School Wellness policies 43% 50%

Farm to School 50% 29%

Farmer’s Markets / Produce Stands 36% 27%

Healthy Celebrations 43% 29%

Safe Routes to School/Active Transportation 14% 29%

Food and Beverage Standards 36% 21%

Healthy Procurement 21% 21%

Access to Free Drinking Water 36% 7%

Healthy Vending Standards 7% 0%

Joint Use Agreements 0% 0% 1No data collected at Baseline.

Page 57 of 61

For Smarter Lunchrooms Movement, half of respondents indicated that youth are involved in

creating artwork promoting menu items and/or taste testing new items before they are added to the

menu. Over a quarter of respondents (29%) indicated that youth are not currently engaged in

Smarter Lunchrooms Movements. However, this is less than the percentage reported by

respondents at baseline (43%).

Table 92. Percentage of respondents that reported youth are involved in the following Smarter Lunchroom Movement activities

(n=14).

%

Youth are involved in creating artwork promoting menu items 50%

Youth are involved in taste testing new items before they are added to the menu 50%

Youth are involved in developing creative and descriptive names for menu items 14%

Youth are involved in modeling healthful eating behaviors to others 14%

Youth participate in lunchroom/school assessments 7%

Youth are not currently engaged 29%

Other (please specify) 21%

Youth-Led Participatory Action Research

Most respondents (64%) indicated that they have experience with YPAR. By comparison, 40% of

respondents who completed the baseline assessment indicated that they had experience with

YPAR. When asked to describe their experience, responses ranged from limited experience (e.g.,

participating in a training) to several years of experience leading YPAR.

Nearly one quarter of respondents (21%) indicated that they are currently engaging youth in a YPAR

project. One respondent shared that their youth will be working on getting their school a healthy

vending machine, and another indicated that the YPAR project will be part of their Smarter

Lunchrooms Movement efforts.

Youth Leadership Opportunities

Most respondents (64%) reported that they are engaging youth in peer-to-peer education or Teens

as Teachers. Some of the respondents described how they engage youth in this way:

“Teaching a nutrition lesson/developing informational boards.” - Supervisor

Graduated chefs teach new chefs cooking skills and nutrition lessons.” - Educator

“4-H Leaders of sustainable you! Day Camp are trained and implement CATCH PE.” – Educator

Two respondents (14%) indicated that they are building youth's capacity to use data and maps and

incorporating data and mapping tools into their UC CalFresh program. In addition, two other

respondents indicated that they plan to begin doing so in the coming year.

Page 58 of 61

Partnerships

Respondents were asked about partners that provide support to their UC CalFresh program,

including the type of support received. The most common partnerships were 4-H and After School

Programs. The most common forms of support included receiving resources or supplies (provided by

five partners), training or capacity building (provided by five partners), and staff support (provided by

four partners). Specific to the most common partnerships:

4-H Program partnerships increased from 57% in the baseline assessment to 85% in the

midpoint. Forms of support included:

o Resources or Supplies (54%)

o Training or Capacity Building (39%)

o Staff Support (59%)

o Youth Recruitment (39%)

o Volunteers (15%)

o Financial Support (31%)

After School Program partnerships remained similar, 64% in the baseline assessment and

62% in the mid-point assessment. Forms of support included:

o Resources or Supplies (15%)

o Training or Capacity Building (8%)

o Staff Support (31%)

o Youth Recruitment (15%)

o Volunteers (15%)

o Financial Support (8%)

Additionally, nearly half of respondents (46%) reported that Dairy Council and Public Health

Departments provide support to their program, 39% reported Boys and Girls Club, 31% reported

County Office of Education, and 23% reported Fuel Up to Play 60.

Many respondents listed other important partnerships for their UC CalFresh program. Common

partners included:

Food banks / food pantries

Master Gardeners

Universities

Parks and Recreation Departments

Department of Social Services

Housing Authority

Community organizations.

Over one-third of respondents (39%) indicated that Community Grants have benefited their UC

CalFresh program. Examples included garden grants, 4-H grants to fund YPAR and physical activity

work, First-5 grants to improve preschool curriculum materials, and a Kaiser Permanente Hydration

Station Grant.

Page 59 of 61

Resources and Training

Respondents 57% indicated that they had attended at least one training listed on the survey (Figure 2).

At least half of respondents indicated they had attended the Data and Mapping training (60%) and/or the

Introduction to YPAR training (50%). All respondents indicated that the trainings they attended had been

helpful. Over one-third of respondents (36%) reported that they are already implementing new strategies

that they learned from having attended training, and another quarter (27%) indicated that they plan to do

so.

Figure 2. Percentage of respondents who indicated they attended the following trainings (n=12).

Respondents indicated that they need additional training in the following areas:

Data and Mapping (86%)

Supporting Youth Voice for Community Change (80%)

Introduction to YPAR (67%)

Peer to Peer (50%)

Supporting Student Action Committees in Creating Healthy Changes in the Lunchroom. (50%)

In addition to the subjects above, respondents provided suggestions for additional training needs,

including:

Social marketing usage and implementation

Youth engagement strategies (non-YPAR)

Use of the Nutrition Voyage curriculum

Working with youth

Asking inquiry questions of youth

22%

10%

50%

20%

60%

Supporting StudentNutrition ActionCommittees in

Creating HealthyChanges in the

Lunchroom

Peer to Peer Introduction toYPAR

Supporting YouthVoice for

Community Change

Data and Mapping

0%

20%

40%

60%

80%

100%

Page 60 of 61

Limitations

For four of the UC CalFresh counties or county clusters, responses were received from two

representatives. Responses often varied by respondent for each UC CalFresh County/County Cluster,

making it difficult to discern what activities are taking place at each UC CalFresh County/County Cluster.

As a result, data was presented in this report by respondent, rather than by UC CalFresh County/County

Cluster, with some UC CalFresh Counties/County Clusters being represented by two respondents.

Frequencies reported in this report may not be fully representative of activities taking place statewide

since all UC CalFresh Counties/County Clusters did not participate in the survey and some counties only

participated in the baseline or the mid-point survey. In addition, because the mid-point Assessment was

completed by Educators in addition to Advisers, Supervisors, and Managers, changes in responses

between the baseline and Mid-Point Survey may be due to changes in the survey sample. Not all results

could be compared to Baseline results due to changes in survey questions.

In some cases, activities reported on the mid-point assessment appeared to have decreased relative to

the baseline assessment. For example, the percentage of respondents that reported youth are initiating

action in High School activities decreased from 33% to 14%. These decreases may be attributed to a few

factors. In addition to possible response changes due to changes in the respondents included in the

sample between baseline and mid-point, it’s possible that changes in awareness and understand among

respondents could have resulted in different responses. For instance, an increase in education about

authentic engagement practices among respondents may have resulted in fewer respondents reporting

that authentic engagement practices are taking place. In the baseline assessment programs may have

over reported engagement levels, but after receiving training in FFY 2016-2017 and reflecting on

activities rendered the midpoint assessment may represent a more accurate reporting of programs that

are authentically allowing youth to lead or initiate action.

RECOMMENDATIONS AND NEXT STEPS

To build upon the momentum of FFY 2016 and 2017, we recommend implementing a Regional Cohort

Model for training and technical assistance for FFY 2018. Youth Engagement Cohorts will include groups

of nutrition educators, supervisors, and/or advisors who are implementing the same youth engagement

program model along the same trajectory. This model will allow for people to come together at strategic

points in the process for collective training and learning, and receive on-going technical assistance and

program development support throughout the process. Each cohort will provide a collaborative,

interactive, and dynamic space for staff and their youth groups to grow their knowledge and skills, share

experiences, support peer mentorship, and foster program sustainability.

In-person cohort gatherings will be regionally-coordinated, topic-specific, in-person workshops for cohort

members to engage in focused training, collaborative project planning, critical reflection, and mutual

support. Participants will commit to sharing ideas, resources, processes, and successes and challenges

about their county’s youth engagement project(s) with each other and the training and technical

assistance team. Trainings will be interspersed throughout the year to assist county programs at key

stages in their youth engagement programs, include the incorporation of data and mapping throughout

their PSE change strategy development and implementation, and consistently utilize youth development

principles and practices. It is intended that youth leaders themselves will be able to attend the final cohort

gathering at the end of the school year. In addition to participating UC CalFresh programs, these cohorts

will be open to all SNAP-Ed Youth Engagement Projects as space allows. Aside from the in-person

gatherings, all other ongoing technical assistance opportunities will continue from FFY 2017—this

includes monthly technical assistance calls and/or meetings with sites, and All-County Youth

Engagement Call six times per year. In between cohort gatherings, participating programs will also be

encouraged to maintain connections with one another to foster stronger relationships and mutual

learning.

Page 61 of 61

Documents

Evaluation Report Attachments #1-5 · FFY 2017 Annual Report Evaluation Attachments Evaluation Report Attachment # 1 Process and Outcome Evaluation: Tracking Policy, Systems,