69
Healthy Schools Healthy Schools Leadership Program Leadership Program (HSLP) (HSLP) Evaluation Plan Evaluation Plan February 2006 February 2006

Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Embed Size (px)

Citation preview

Page 1: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Healthy Schools Healthy Schools Leadership Program Leadership Program

(HSLP)(HSLP)

Evaluation PlanEvaluation PlanFebruary 2006February 2006

Page 2: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

What I’m going to talk What I’m going to talk aboutabout

Present & get your feedback on Present & get your feedback on Evaluation Plan for HSLP 06-08 cohortEvaluation Plan for HSLP 06-08 cohort

Generate ideas for your own Generate ideas for your own assessment/evaluation workassessment/evaluation work

Ideas for using the Healthy Youth SurveyIdeas for using the Healthy Youth Survey

Please consider this a discussion!Please consider this a discussion!

Page 3: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

““George tried to be a good George tried to be a good monkey…”monkey…”

Page 4: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

The Primary Evaluation The Primary Evaluation QuestionQuestion

Is the Healthy Schools Leadership Program Is the Healthy Schools Leadership Program (HSLP) effective for making schools (HSLP) effective for making schools healthier?healthier?

Note that this is a state-level evaluation Note that this is a state-level evaluation questionquestion

I will present an ambitious scenario for I will present an ambitious scenario for evaluationevaluation

We will attempt to be as rigorous as possibleWe will attempt to be as rigorous as possible I’m not a schools expert… you are the schools I’m not a schools expert… you are the schools

expertsexperts

Page 5: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Caveats to Evaluation Caveats to Evaluation ModelModel

Small numbers of schools (n=15)Small numbers of schools (n=15) Change takes time – especially for long-Change takes time – especially for long-

term outcomesterm outcomes Our comparison group is the rest of the Our comparison group is the rest of the

statestate If findings are positive, some will argue If findings are positive, some will argue

that this is the results of your existing that this is the results of your existing capacity/readinesscapacity/readiness

The plan is ambitious – looking for The plan is ambitious – looking for additional resources to support evaluation additional resources to support evaluation

Page 6: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

What we can say right now with What we can say right now with certainty is that if you spend certainty is that if you spend resources implementing resources implementing coordinated school health it coordinated school health it doesn’t appear to make your doesn’t appear to make your academic achievement any academic achievement any worse….worse….

I’d like to say something I’d like to say something strongerstronger

Page 7: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Describe the ProgramDescribe the Program

Page 8: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Coordinated School Coordinated School Health GoalsHealth Goals

Health EducationHealth Education Physical EducationPhysical Education Counseling/Support ServicesCounseling/Support Services Food/Nutrition ServicesFood/Nutrition Services Healthy EnvironmentHealthy Environment Health ServicesHealth Services Family/Community InvolvementFamily/Community Involvement Staff WellnessStaff Wellness

Page 9: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Simplified Logic ModelSimplified Logic ModelB.

Activities

Ass’tPlan

Evaluate

A. Inputs

Teams

HSLP training

C. Outputs

Ass’t results

Action Plan

E. Intermediate Outcomes

Policy, procedure,

environment changes

Sustainability

D. Short-term Outcomes

Results from action plan activity

F. Long-term Outcomes

Staff/studn’t behavior change

G. ImpactsHealth

improvem’tAcademic achievem’t

Catalysts

Barriers

Non-HSLP External Factors

Page 10: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Getting the Right Getting the Right Perspective: Evaluation Perspective: Evaluation

QuestionsQuestions

Page 11: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Specific AimSpecific Aim

1. Describe the extent to which the inputs and 1. Describe the extent to which the inputs and activities are sufficient to generate outputs – activities are sufficient to generate outputs – assessment results, action plans (A/Bassessment results, action plans (A/BC, C, Descriptive)Descriptive)

Were teams able to assemble wellness committees? How did Were teams able to assemble wellness committees? How did they function? Were they able to complete assessment and they function? Were they able to complete assessment and planning? What priorities were identified? What were the planning? What priorities were identified? What were the barriers and motivating factors related to output barriers and motivating factors related to output completion?completion?

Are there differences among HSLP teams in satisfaction Are there differences among HSLP teams in satisfaction with outputs? What are the factors (HSLP or other) with outputs? What are the factors (HSLP or other) associated with high/low satisfaction? Was HSLP a critical associated with high/low satisfaction? Was HSLP a critical factor in achievements of high-quality outputs?factor in achievements of high-quality outputs?

Data Sources: Team interviews, action plans, Healthy Data Sources: Team interviews, action plans, Healthy Schools Report CardSchools Report Card

Page 12: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Hypothetical FindingsHypothetical Findings All schools assembled teams; X% All schools assembled teams; X%

included community membersincluded community members 12 of 15 teams created formal action 12 of 15 teams created formal action

plansplans All schools identified at least one All schools identified at least one

nutrition priority nutrition priority School teams that were “very satisfied” School teams that were “very satisfied”

with their plans were more likely to have with their plans were more likely to have involved community members and had a involved community members and had a clear process for translating assessment clear process for translating assessment results into an action planresults into an action plan

Page 13: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Specific AimSpecific Aim2. Describe the extent to which participation in the 2. Describe the extent to which participation in the

HSLP is associated with short-term outcomes – HSLP is associated with short-term outcomes – early results of action plans (A/Bearly results of action plans (A/BD, Descriptive)D, Descriptive)

Were school wellness teams able to use plans to initiate Were school wellness teams able to use plans to initiate change? How did implementation activity begin? Which change? How did implementation activity begin? Which priorities did schools take action on first? How were priorities did schools take action on first? How were responsibilities identified? What factors were associated responsibilities identified? What factors were associated with transition from planning to action?with transition from planning to action?

Are there differences among HSLP schools in their Are there differences among HSLP schools in their satisfaction with/quality of early implementation? Were satisfaction with/quality of early implementation? Were some kinds of activities more easily translated to quality some kinds of activities more easily translated to quality action? What are the factors (HSLP or other) associated action? What are the factors (HSLP or other) associated with high/low satisfaction? Was HSLP a critical factor in with high/low satisfaction? Was HSLP a critical factor in initiating high-quality work?initiating high-quality work?

Data Sources: Team interviews, action plan reports, Data Sources: Team interviews, action plan reports, Healthy Schools Report CardHealthy Schools Report Card

Page 14: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Hypothetical FindingsHypothetical Findings

10 of 15 schools increased the 10 of 15 schools increased the overall number of “maintenance” overall number of “maintenance” items on their Healthy Schools items on their Healthy Schools Report CardReport Card

Teams reported highest satisfaction Teams reported highest satisfaction with their work on XXX prioritieswith their work on XXX priorities

XX Teams (X%) said they “strongly XX Teams (X%) said they “strongly agreed” that change would not have agreed” that change would not have occurred without the HSLP training occurred without the HSLP training and technical assistanceand technical assistance

Page 15: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Specific AimSpecific Aim3. Describe the extent to which schools that 3. Describe the extent to which schools that

participated in HSLP change their environments participated in HSLP change their environments (A/B(A/BE, Pre/post comparison to non-HSLP schools)E, Pre/post comparison to non-HSLP schools)

Did HSLP school environments change, and were changes Did HSLP school environments change, and were changes different than those of non-participating schools? Which different than those of non-participating schools? Which changes were achieved?changes were achieved?

Are there differences among HSLP schools that achieved Are there differences among HSLP schools that achieved policy/procedure/environment change vs. those that did policy/procedure/environment change vs. those that did not? not?

How has sustainability been addressed? What additional How has sustainability been addressed? What additional resources have been attracted for school health? Has resources have been attracted for school health? Has institutionalization of school health work been achieved?institutionalization of school health work been achieved?

Data Sources: SHEP, Healthy Schools Report Card, Data Sources: SHEP, Healthy Schools Report Card, HYS (perceived school environments), Staff HYS (perceived school environments), Staff wellness survey (if perceived environment wellness survey (if perceived environment indicators included)indicators included)

Page 16: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Sample FindingsSample FindingsEducate Staff about Asthma

0%

20%

40%

60%

80%

100%

Was

hing

ton

Stat

e to

tal

Non

-Ste

psAr

eas

Step

sAr

eas

Cla

rkC

ount

yC

hela

n-D

ougl

as-

Oka

noga

nTh

urst

onC

ount

y

Col

ville

Trib

e

2004

** significantp=.04

Source: SHEP 2004

Page 17: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Specific AimSpecific Aim

4. Describe the extent to which schools that 4. Describe the extent to which schools that participated in HSLP change their student/staff participated in HSLP change their student/staff behaviors (A/B behaviors (A/B F, Pre/post comparison to F, Pre/post comparison to non-HSLP schools for students, self-non-HSLP schools for students, self-comparison for staff)comparison for staff)

Did HSLP schools’ student behaviors change, and Did HSLP schools’ student behaviors change, and were changes different than those of non-were changes different than those of non-participating schools? participating schools?

Did HSLP schools’ staff behaviors change? Did HSLP schools’ staff behaviors change? Are there differences among HSLP schools that Are there differences among HSLP schools that

achieved behavior change vs. those that did not?achieved behavior change vs. those that did not?

Data Sources: HYS, Staff Wellness SurveyData Sources: HYS, Staff Wellness Survey

Page 18: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Sample FindingsSample FindingsMeet Recommendations Vigorous Exercise

10th Grade

0%

30%

60%

90%

Was

hing

ton

Stat

e to

tal

Non

-Ste

psAr

eas

Step

sAr

eas

Cla

rkC

ount

yC

hela

n-D

ougl

as-

Oka

noga

nTh

urst

onC

ount

y

Col

ville

Trib

e

2002 2004

Source: Healthy Youth Survey 2002-2004

Page 19: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Specific AimSpecific Aim

5. Describe the extent to which schools that 5. Describe the extent to which schools that participated in HSLP have had health impact (A/B participated in HSLP have had health impact (A/B G, Pre/post comparison to non-HSLP schools G, Pre/post comparison to non-HSLP schools for WASL, self-comparison for absenteeism)for WASL, self-comparison for absenteeism)

Did HSLP schools’ student academic achievement Did HSLP schools’ student academic achievement change, and were changes different than those of non-change, and were changes different than those of non-participating schools?participating schools?

Did HSLP schools’ student and/or staff absenteeism Did HSLP schools’ student and/or staff absenteeism rates change?rates change?

Are there differences among HSLP schools that achieved Are there differences among HSLP schools that achieved impact vs. those that did not? impact vs. those that did not?

Data Sources: WASL, Student absenteeism measures, Data Sources: WASL, Student absenteeism measures, Staff absenteeism measuresStaff absenteeism measures

Page 20: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Sample FindingsSample FindingsMiddle School WASL Math

1999-00 vs. 2003-04 (Source: OSPI)

0%

10%

20%

30%

40%

50%

60%

1999-00 2003-04

Math

Skyridge Kulshan Eisenhower State 7th AYP Target

Page 21: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Data Sources: Data Sources: Gathering Credible Gathering Credible EvidenceEvidence

“Not everything that counts can be counted; and not everything that can be counted counts.”

— Albert Einstein

Page 22: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Data SourcesData Sources Proposing to use data you already Proposing to use data you already

collect, or new data collection related to collect, or new data collection related to your HSLP implementationyour HSLP implementation Exception: interviews of Exception: interviews of

team/administrationteam/administration For anything where we want to look at For anything where we want to look at

everyone together, methods have to be everyone together, methods have to be standard…standard…

For some things it would be fine to have For some things it would be fine to have sentinel schools sentinel schools See school-specific data suggestions later See school-specific data suggestions later

Page 23: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Team InterviewsTeam Interviews

During Spring 06 develop interview During Spring 06 develop interview protocols for assessing function and protocols for assessing function and satisfaction among HSLP participants (see satisfaction among HSLP participants (see evaluation questions)evaluation questions)

Repeat in Spring 08 Repeat in Spring 08 Augment with interviews with other Augment with interviews with other

wellness committee members (or lead wellness committee members (or lead administrators?)administrators?)

Your role:Your role: participate in interviews participate in interviews

Page 24: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Healthy Schools Report Healthy Schools Report CardCard

Conduct assessment to identify priorities Conduct assessment to identify priorities for action planningfor action planning

ASCD (Association for Supervision and ASCD (Association for Supervision and Curriculum Development) has a clear Curriculum Development) has a clear process/instrumentprocess/instrument

Your role:Your role: use good assessment to use good assessment to develop plans, provide a copy of whatever develop plans, provide a copy of whatever baseline (spring 06) and follow-up (spring baseline (spring 06) and follow-up (spring 08) assessment results08) assessment results

Page 25: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

HSRC Component AreasHSRC Component Areas

1.1. School health program policy & strategic School health program policy & strategic planningplanning

2.2. Coordination of school health programsCoordination of school health programs3.3. Social and emotional climateSocial and emotional climate4.4. Family and community involvementFamily and community involvement5.5. School facilities and transportationSchool facilities and transportation6.6. Health educationHealth education7.7. Physical education and physical activityPhysical education and physical activity8.8. Food and nutrition servicesFood and nutrition services9.9. School health servicesSchool health services10.10. Counseling, psychological and social work Counseling, psychological and social work

servicesservices11.11. School-site health promotion for staffSchool-site health promotion for staff

Page 26: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

• GreenGreen = short-term high priority = short-term high priority• RedRed = short-term lower priority = short-term lower priority• LimeLime = long-term, highest = long-term, highest

prioritypriority• BlueBlue = long-term, high priority = long-term, high priority• OrangeOrange = long-term, lower priority = long-term, lower priority• No color = MaintenanceNo color = Maintenance

HSRC Indicator Priority HSRC Indicator Priority RankingsRankings

Multiple specific topics within Multiple specific topics within componentscomponents

Rate with your team for each: Rate with your team for each: current status, perceived benefit, current status, perceived benefit, perceived effortperceived effort

Color-coded result report for Color-coded result report for combined scorescombined scores

Page 27: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

HSRC Sample HSRC Sample Component-TopicComponent-Topic

School health program policy & School health program policy & strategic planningstrategic planning

All staff members are provided with All staff members are provided with time/resources to comply with health time/resources to comply with health program policiesprogram policies

Coordination of school health programsCoordination of school health programs School health program members regularly School health program members regularly

inform the principal, district inform the principal, district superintendent, or school board of current superintendent, or school board of current developments in the school health program.developments in the school health program.

Page 28: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Annual Action Plans & Annual Action Plans & ReportsReports

Design your action plan based on Design your action plan based on assessment resultsassessment results

Your role:Your role: provide action plan provide action plan annually, report of progress annually annually, report of progress annually (format?) (format?)

Page 29: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

School Health Education School Health Education Profile (SHEP)Profile (SHEP)

Survey of Secondary School Survey of Secondary School Principals and Lead Health TeachersPrincipals and Lead Health Teachers

Sponsored by CDCSponsored by CDC Will be sent in Spring 06 and Spring Will be sent in Spring 06 and Spring

0808

Your role:Your role: encourage completion of encourage completion of the surveythe survey

Page 30: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

SHEP/Profiles Data SHEP/Profiles Data Examples:Examples:

School Health Advisory School Health Advisory GroupsGroups Have a School Health Advisory Have a School Health Advisory

Group Group 59% of high schools59% of high schools 56% of middle schools56% of middle schools 46% of combined secondary schools46% of combined secondary schools 55% overall55% overall Similar to 52% total in 2002Similar to 52% total in 2002

Page 31: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

SHEP/Profiles Data SHEP/Profiles Data Examples:Examples:TobaccoTobacco

A designated individual has primary A designated individual has primary responsibility for seeing that the responsibility for seeing that the tobacco prevention policy is enforcedtobacco prevention policy is enforced 64% high school64% high school 68% middle school68% middle school 59% secondary combined59% secondary combined 65% overall65% overall Similar to 68% overall in 2002Similar to 68% overall in 2002

No association with school advisory No association with school advisory presencepresence

Page 32: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

SHEP/Profiles Data SHEP/Profiles Data Examples:Examples:NutritionNutrition

Students can purchase fresh Students can purchase fresh fruits/vegetables from vending fruits/vegetables from vending machines or at the school store, machines or at the school store, canteen, or snack barcanteen, or snack bar 49% high school49% high school 45% middle school45% middle school 39% secondary combined39% secondary combined 46% overall46% overall Slight improvement from 39% overall in 2002Slight improvement from 39% overall in 2002

No association with school advisory presenceNo association with school advisory presence

Page 33: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

SHEP/Profiles Data SHEP/Profiles Data Examples:Examples:

AsthmaAsthma Obtain and use an Asthma Action Plan Obtain and use an Asthma Action Plan

(or Individualized Health Plan) for all (or Individualized Health Plan) for all students with asthmastudents with asthma 77% high school77% high school 77% middle school77% middle school 74% secondary combined74% secondary combined 76% overall76% overall Improvement from 62% overall in 2002Improvement from 62% overall in 2002

Schools with advisories are twice as likely to Schools with advisories are twice as likely to obtain asthma plans as schools without obtain asthma plans as schools without advisoriesadvisories

Page 34: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Staff Wellness SurveyStaff Wellness Survey Designed by CDC – available in MarchDesigned by CDC – available in March Assuming we can modify (shorten) if neededAssuming we can modify (shorten) if needed Assuming we (DOH/OSPI) can provide as Assuming we (DOH/OSPI) can provide as

online survey and give data back to school online survey and give data back to school teamsteams

Are there other ways to measure staff Are there other ways to measure staff absences?absences?

Your role:Your role: engage staff to participate engage staff to participate Caution: poor participation will not yield Caution: poor participation will not yield

useful data!useful data!

Page 35: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Healthy Youth Survey Healthy Youth Survey (HYS)(HYS)

Administered Fall of even yearsAdministered Fall of even years Grades 6,8,10,12Grades 6,8,10,12 How many HSLP schools participated How many HSLP schools participated

in 2002, 2004?in 2002, 2004? Most powerful comparison will be 02-Most powerful comparison will be 02-

04-0604-06

Your role:Your role: register to participate in register to participate in 2006 (soon!)2006 (soon!)

Page 36: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Student AbsenteeismStudent Absenteeism

NCLB requires measurement of NCLB requires measurement of excused/unexcused absencesexcused/unexcused absences Assuming no change in non-illness Assuming no change in non-illness

excused absences, we can monitor excused absences, we can monitor changes in this rate over timechanges in this rate over time

Your role:Your role: provide annual absence provide annual absence ratesrates

Page 37: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Academic Academic Achievement/WASLAchievement/WASL

Annual, building-levelAnnual, building-level Grades 4,7,10Grades 4,7,10 Math, Reading, WritingMath, Reading, Writing

Your role:Your role: none needed none needed

Page 38: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Other School FactorsOther School Factors Free & reduced lunch enrollment [this may Free & reduced lunch enrollment [this may

be an outcome, consider different measure of be an outcome, consider different measure of SES]SES]

Percent minority enrollmentPercent minority enrollment School/district size/enrollmentSchool/district size/enrollment School level (elementary, middle, high, School level (elementary, middle, high,

combination)combination) Urban/rural locationUrban/rural location Baseline levels of outcomesBaseline levels of outcomes Other (baseline) funding sources for health Other (baseline) funding sources for health

improvementimprovement

Your role:Your role: none – we can obtain none – we can obtain

Page 39: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Too much information?Too much information?

Page 40: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

TimelineTimeline

J-J 04 Jn-D 04 J-J 05 Jl-D 05 J-J 06 Jl-D 06 J-J 07 Jl-D 07 J-J 08 Jl-D 08 J-J 09 Jl-D 09

SHEP 04 SHEP 06 SHEP 08 HYS 04 HYS 06 HYS 08 Healthy Schools Leadership Institute

Team Interv iews Team Interv iews

Healthy Schools Report Card Staff Wellness Survey

Healthy Schools Report Card Staff Wellness Survey

Evaluation Outcomes Report

Page 41: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

School-level EvaluationSchool-level Evaluation

We won’t evaluate you as individual We won’t evaluate you as individual teamsteams

Evaluating yourself on a few points Evaluating yourself on a few points would be helpful in telling your story would be helpful in telling your story laterlater

Suggest designating a lead for local Suggest designating a lead for local evaluation (just like any evaluation (just like any implementation piece)implementation piece)

Page 42: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Sample School-level Sample School-level Evaluation DataEvaluation Data

vending revenue datavending revenue data key informant interviews with students or staffkey informant interviews with students or staff tracking of policy enforcement activitiestracking of policy enforcement activities other data gathering from communities other data gathering from communities health room utilization datahealth room utilization data student absences – use more advanced student absences – use more advanced

tracking system (Public Health Tracking tracking system (Public Health Tracking System grant available from DOH System grant available from DOH environmental health)environmental health)

lots of (digital) pictures of your environment – lots of (digital) pictures of your environment – now and later now and later

Page 43: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Still Awake??Still Awake??

Page 44: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Healthy Youth Healthy Youth SurveySurvey

Page 45: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Why do I need HYS now?Why do I need HYS now?

Contribute to assessment, planningContribute to assessment, planning Identification of your prioritiesIdentification of your priorities Describing affected studentsDescribing affected students

Gain attention for student health as Gain attention for student health as a prioritya priority

Page 46: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

HYS 2004: Collaborative Effort of HYS 2004: Collaborative Effort of OSPI, DOH, DSHS, CTED, FPC, and OSPI, DOH, DSHS, CTED, FPC, and

RMC ResearchRMC Research State-level simple random sample of schoolsState-level simple random sample of schools

County-level samples drawn as appropriate County-level samples drawn as appropriate Non-sampled schools also invited to participateNon-sampled schools also invited to participate Survey consistently administered in the Fall of Survey consistently administered in the Fall of

even yearseven years Survey given to 6Survey given to 6thth, 8, 8thth, 10, 10thth, and 12, and 12thth graders graders Survey booklets have one-page tear-off answer Survey booklets have one-page tear-off answer

sheetsheet Survey for 8Survey for 8thth, 10, 10thth, and 12, and 12thth graders uses 2 graders uses 2

form “interleaved” administrationform “interleaved” administration

Page 47: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

The Healthy Youth Survey 2004 was The Healthy Youth Survey 2004 was completed… completed…

HYS 2004: ParticipationHYS 2004: Participation

By 185,095 students…

In 1,013 schools…

In 235 school districts…

In all 39 counties.

Page 48: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Risk & Protective Factor Risk & Protective Factor DataData

Consistent with CTC & MTF SurveysConsistent with CTC & MTF SurveysDevelopment led by DSHS/DASA & OSPIDevelopment led by DSHS/DASA & OSPI

WSSAHB Questionnaire• Demographics• Alcohol, Tobacco, other drugs• School risk/protective factors• Community risk/protective factors• Peer-Individual-Family risk/ protective factors

Page 49: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Youth Health Behavior Youth Health Behavior DataData

Consistent with National YRBS & YTSConsistent with National YRBS & YTSDevelopment led by DOHDevelopment led by DOH

YRBS Questionnaire• Demographics• Alcohol, Tobacco, Other drugs• Nutrition• Physical Activity• Safety Behaviors (Helmets, Seatbelts)• Mental Health/Depression/Suicide• Additional Tobacco Indicators

Page 50: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Getting the Best of Both Getting the Best of Both

Form AWSSAHB-like

Form BYRBS-likeSurvey

Core

Page 51: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

66thth Grade Survey Grade Survey

One Form

SurveyCore

+ Mutually Agreed Items

Page 52: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Core Survey ItemsCore Survey Items

DemographicsDemographics Alcohol, Tobacco and Other Drug Alcohol, Tobacco and Other Drug

Use ItemsUse Items Key Violence-Related ItemsKey Violence-Related Items School-Specific Asset ItemsSchool-Specific Asset Items DepressionDepression

Page 53: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Administration Administration proceduresprocedures

Safe and confidential environmentSafe and confidential environment Standardized administration procedures (e.g., Standardized administration procedures (e.g.,

coordinator training, teacher training, written coordinator training, teacher training, written instructions, teacher stays in room but at desk, instructions, teacher stays in room but at desk, single class period to avoid discussion, absent single class period to avoid discussion, absent students do not make up)students do not make up)

Students informed of importance of surveyStudents informed of importance of survey No student name or other identifying No student name or other identifying

informationinformation Students place own answer sheet in envelopeStudents place own answer sheet in envelope Students provided resource listStudents provided resource list

Page 54: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Data Cleaning Data Cleaning ProceduresProcedures

Inconsistent answersInconsistent answers Evidence of faking high level of Evidence of faking high level of

substance usesubstance use DishonestyDishonesty Wrong gradeWrong grade

Page 55: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

What is a Confidence Interval?

• The reported value is probably a little different than the “true” value for all your students

• We are “95% confident” that the “true” value is within the +/- range, called the “confidence interval”

Page 56: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Interpreting CIs Interpreting CIs GraphicallyGraphically

0 5 10 15 20 25 30

Smokedcigarettes

Percent of students

Page 57: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Local Report Contents

• Introduction and Overview• Highlights of Local Results• Selected Results by Gender• Responses to Items• Risk and Protective Factors• List of Core Items & Item index

Page 58: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

For your continued For your continued exploration…exploration…

http://http://www.rmccorp.comwww.rmccorp.com/HYS06/index.htm/HYS06/index.htm

http://www3.doh.wa.gov/hys/http://www3.doh.wa.gov/hys/

Page 59: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Academic Risk by Asthma Status

26.6% 25.4% 24.5% 26.7%

40.5%

0%

10%

20%

30%

40%

50%

No Asthma Mild Intermittent Mild Persistent ModeratePersistent

SeverePersistent

Aca

dem

ic R

isk

Source: 2004 Healthy Youth Survey, state sample only. Graph uses unadjusted estimates; odds ratios are for estimates adjusted for grade (8-10-12) and maternal education (3-level).

• Students with severe persistent asthma symptoms had increased odds for academic risk

Page 60: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Academic Risk by Diabetes Status

25.5%

37.6%

0%

10%

20%

30%

40%

50%

No Diabetes Diabetes

Aca

dem

ic R

isk

Source: 2004 Healthy Youth Survey, state sample only. Graph uses unadjusted estimates; odds ratios are for estimates adjusted for grade (8-10-12) and maternal education (3-level).

• Students with diabetes had increased odds for academic risk

Page 61: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Academic Risk by Cigarette Smoking

24.0%

49.7%

0%

10%

20%

30%

40%

50%

Not Smoking Smoking

Aca

dem

ic R

isk

Source: 2004 Healthy Youth Survey, state sample only. Graph uses unadjusted estimates; odds ratios are for estimates adjusted for grade (8-10-12) and maternal education (3-level).

• Students who smoked any cigarettes during the past monthhad increased odds for academic risk

Page 62: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Healthy Youth Survey Healthy Youth Survey WebsiteWebsite

Some quick crosstabs examples you could Some quick crosstabs examples you could run…run… Kids who smoke are less likely to report having Kids who smoke are less likely to report having

access to an ATOD counselor (61% among access to an ATOD counselor (61% among smokers v. 71% among non-smokers, 10smokers v. 71% among non-smokers, 10thth grade grade state sample)state sample)

Kids who report being bullied are also more likely Kids who report being bullied are also more likely to report depression (46% depression among to report depression (46% depression among bullied v. 29% among non-bullied, 10bullied v. 29% among non-bullied, 10thth grade) grade)

Kids who drank more than 2 soda pops per day Kids who drank more than 2 soda pops per day were more likely to report getting Ds and Fs (7% were more likely to report getting Ds and Fs (7% vs. 15% among heavy pop-drinkers, 10vs. 15% among heavy pop-drinkers, 10thth grade) grade)

Page 63: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Healthy Youth Survey 2006 Healthy Youth Survey 2006 – –

HSLP related measuresHSLP related measures Physical EducationPhysical Education

Number of days of PE per weekNumber of days of PE per week How long doing actual activity in PE classHow long doing actual activity in PE class

Counseling/Student Support ServicesCounseling/Student Support Services Is there a school counselor to talk about ATODIs there a school counselor to talk about ATOD Is there someone you could go to for helpIs there someone you could go to for help

Food/Nutrition ServicesFood/Nutrition Services New junk food module in 2006 asks about soda New junk food module in 2006 asks about soda

pop and junk/fried food consumption at school, pop and junk/fried food consumption at school, also breakfastalso breakfast

Fruit & vegetable “5-a-day” consumptionFruit & vegetable “5-a-day” consumption Soda pops per day (previous day)Soda pops per day (previous day)

Page 64: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

HYS 06 – HSLP-related HYS 06 – HSLP-related measuresmeasures Healthy EnvironmentHealthy Environment

Feeling safe at schoolFeeling safe at school Enjoy being in school/hate being in schoolEnjoy being in school/hate being in school Carry weapon on school propertyCarry weapon on school property BullyingBullying Teachers care about meTeachers care about me Using tobacco on school propertyUsing tobacco on school property Been drunk or high at schoolBeen drunk or high at school Tobacco rules are usually enforcedTobacco rules are usually enforced

Health EducationHealth Education Number of times received tobacco educationNumber of times received tobacco education Number of times practices ways to say “no” Number of times practices ways to say “no” How good in school at educating about HIVHow good in school at educating about HIV

Page 65: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

HYS 06 – HSLP-Related HYS 06 – HSLP-Related MeasuresMeasures

Family/Community InvolvementFamily/Community Involvement Dinner with family at homeDinner with family at home Parent/guardian discussed tobacco at Parent/guardian discussed tobacco at

homehome How easy to get tobacco/alcohol if you How easy to get tobacco/alcohol if you

wanted somewanted some Student Support ServicesStudent Support Services

Ever had an asthma action planEver had an asthma action plan

Page 66: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Your Questions…Your Questions…

What if I can’t find my school or What if I can’t find my school or district report?district report?

What if I just need some general help What if I just need some general help getting started with HYS data or the getting started with HYS data or the website?website?

What if I want more analysis done What if I want more analysis done with our data? with our data?

What if I have specific questions What if I have specific questions about HYS data and our Health about HYS data and our Health Department staff can’t help me?Department staff can’t help me?

Page 67: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

You made it!You made it!

Page 68: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

What we talked about…What we talked about…

Present & get your feedback on Present & get your feedback on Evaluation Plan for HSLP 06-08 Evaluation Plan for HSLP 06-08 cohortcohort

Generate ideas for your own Generate ideas for your own assessment/evaluation workassessment/evaluation work

Ideas for using the Healthy Youth Ideas for using the Healthy Youth SurveySurvey

Page 69: Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

Here to help!Here to help!

General QuestionsGeneral Questions

Lori, Martin, PamLori, Martin, Pam

HSLP Evaluation Detail HSLP Evaluation Detail

Julia Dilley 360-402-7877Julia Dilley 360-402-7877

[email protected]@state.or.us

Healthy Youth Survey Healthy Youth Survey

Diane Pilkey 360-236-Diane Pilkey 360-236-35263526

[email protected]@doh.wa.gov