Upload
anon-232666
View
221
Download
0
Embed Size (px)
Citation preview
8/14/2019 description: tags: REL 2007034 body
1/22
I S S U E S& AN SWER S
U . S . D e p a r t m e n t o f E d u c a t i o n
Measuringresilience
and youthdevelopment:the psychometricproperties of
the HealthyKids Survey
R E L 2 0 0 7 N o . 0 3 4
At WestEd
8/14/2019 description: tags: REL 2007034 body
2/22
Measuring resilience and youthdevelopment: the psychometric
properties of the Healthy Kids SurveySeptember 2007
Prepared by
Thomas L. HansonRegional Educational Laboratory West
Jin-Ok KimRegional Educational Laboratory West
I S S U E S&ANSWERS R E L 2 0 0 7 N o . 0 3 4
U . S . D e p a r t m e n t o f E d u c a t i o n
At WestEd
8/14/2019 description: tags: REL 2007034 body
3/22
Issues & Answers is an ongoing series o reports rom short-term Fast Response Projects conducted by the regional educa-
tional laboratories on current education issues o importance at local, state, and regional levels. Fast Response Project topicschange to reect new issues, as identied through lab outreach and requests or assistance rom policymakers and educa-
tors at state and local levels and rom communities, businesses, parents, amilies, and youth. All Issues & Answers reports
meet Institute o Education Sciences standards or scientically valid research.
September 2007
Tis report was prepared or the Institute o Education Sciences (IES) under Contract ED-06-CO-0014 by Regional Edu-
cational Laboratory West administered by WestEd. Te content o the publication does not necessarily reect the views or
policies o IES or the U.S. Department o Education nor does mention o trade names, commercial products, or organiza-
tions imply endorsement by the U.S. Government.
Tis report is in the public domain. While permission to reprint this publication is not necessary, it should be cited as:
Hanson, . L., & Kim, J. O. (2007).Measuring resilience and youth development: the psychometric properties o the Healthy
Kids Survey. (Issues & Answers Report, REL 2007No. 034). Washington, DC: U.S. Department o Education, Institute o
Education Sciences, National Center or Education Evaluation and Regional Assistance, Regional Educational Laboratory
West. Retrieved rom http://ies.ed.gov/ncee/edlabs
Tis report is available on the regional educational laboratory web site at http://ies.ed.gov/ncee/edlabs.
WA
OR
ID
MT
NV
CAUT
AZ
WY
ND
SD
NE
KS
CO
NM
TX
OK
CO
AR
LA
MSAL GA
SC
NC
VAWV
KY
TN
PA
NY
FL
AK
MN
WI
IA
IL IN
MI
OH
VT
NH
ME
MO
At WestEd
8/14/2019 description: tags: REL 2007034 body
4/22
Summary
This report summarizes fndings rom a
study o the psychometric properties o
the resilience and youth development
module, a key component o the Healthy
Kids Survey. The study aims to improve
resilience assessment and research so
that educators can shape the school envi-
ronment to promote academic resilience.
Te Healthy Kids Survey (HKS) is a compre-
hensive student sel-report tool or monitoring
the school environment and student health
risks. Tis report ocuses on one module o the
survey, the resilience and youth development
module (RYDM), which assesses environmen-tal and internal assets associated with posi-
tive youth development and school success.
Environmental assets reer to meaningul and
pro-social bonding to community, school,
amily, and peers. Internal assets are personal
resilience traits, such as sel-ecacy and
problem-solving skills
A part o the resilience and youth development
module is administered to 600,000 studentsin Caliornia every year. School districts
and schools, which receive both single-year
prevalence data and trend data gathered by
the module, use the data to evaluate their local
programs and guide decisionmaking. Te
Healthy Kids Survey and the resilience and
youth development module were designed as an
epidemiological surveillance tool to track ag-
gregate levels o health risk and resilience. Te
module increasingly is being used in evaluation
work to assess student-level changes over time.
However, widespread use o the module,
particularly or evaluation, may be premature.
Te psychometric properties o specic scales
assessed by the elementary school module
have yet to be established. Te secondary
school module has not been validated since
2000, when the instrument was rst tested in
the eld. Te instrument has since undergone
several modications, however, and must be re-
validated. Moreover, measurement equivalenceacross dierent grades, males and emales, and
racial and ethnic groups has never been exam-
ined. Given Caliornias diversity, demonstrat-
ing the cultural appropriateness o the module
or dierent racial and ethnic groups is critical.
Using HKS data processed or school districts
by WestEds Health and Human Development
Program, Regional Educational Laboratory
West analyzed the modules psychometricproperties. Tis report describes the results o
this analysis, provides recommendations on
the proper use o the instrument, and suggests
modications to the module.
For the secondary school module, the results
are consistent with the instruments current
Ms s th
mt: th shmtts th Hth Ks S
8/14/2019 description: tags: REL 2007034 body
5/22
Summary
use as an epidemiological tool and with its
conceptual oundation. It provides compre-
hensive and balanced coverage o eight envi-
ronmental resilience assets and our internal
resilience assets; its subscales exhibit goodinternal consistency and are associated with
student risk actors in expected ways. And i
certain items are dropped, the module also
demonstrates measurement equivalence across
racial/ethnic groups, males and emales, and
grades. Te secondary school RYDM scales ex-
hibit low test-retest reliability, however, which
suggests that the module is not well suited or
examining student-level changes over time.
Te instrument was not designed to examineindividual dierences across students and
should not be used this way. Moreover, two
o the six internal assets that the secondary
school module was designed to measure
cooperation and goals/aspirationscould not
be assessed validly. Several measures would
benet i additional items were included in
derived scales to increase domain coverage.
Te elementary school module was designed
to assess seven environmental resilience assets
and three internal resilience assets, but it can
reliably assess only two environmental as-
sets and one internal asset. Most o the scales
measured by the elementary school instru-
ment have poor psychometric properties. Te
elementary school instrument should thus be
modied considerably to make it suitable orresearch.
September 2007
8/14/2019 description: tags: REL 2007034 body
6/22
Table o conTenTS
Why this study? 1
Developing a risk and resilience assessment tool 2Te Healthy Kids Surveyassessing risk and protective actors 2
Te resilience and youth development moduleassessing the other side o risk 4
Evaluating the psychometric properties o the resilience and youth development module 8Results o the analysis o the secondary school module 10Results o the analysis o the elementary school module 11
Recommendations 12Secondary school environmental resilience assets 12Secondary school internal resilience assets 12Elementary school environmental and internal assets 13
Appendix A Analytic strategy 15
Appendix B Results 21
Appendix C Results and model selection details 44
Appendix D Other assessments o resilience and related actors 53
Appendix E Detailed tables 55
Notes 164
Reerences 165
Boxes
1 Specications o the Healthy Kids Survey 32 Data and analytic strategies 9
Figures
1 Conceptual model or the resilience and youth development module 5
A1 Hypothetical example o MIMIC approach or testing or measurement equivalence 18
C1 Secondary environmental resilience asset scree plot, total analytic samples 44
C2 Elementary school environmental resilience asset scree plot, total analytic samples 46
C3 Secondary school internal resilience asset scree plot, total analytic samples 50
C4 Elementary school internal resilience asset scree plot, total analytic samples 52
Tables
1 Items on the secondary school resilience and youth development module by construct, 2006/07 6
2 Elementary school resilience and youth development module items by construct, 2006/07 8
3 Recommended measures o environmental resilience assets among secondary school students 13
4 Recommended measures o internal resilience assets among secondary school students 14
8/14/2019 description: tags: REL 2007034 body
7/22
5 Recommended measures o environmental and internal resilience assets among elementary schoolstudents 14
A1 Missing data patterns or secondary and elementary samples rom the resilience and youth developmentmodule 16
B1 Secondary school environmental resilience asset exploratory actor analysis results, main sample, 8-actorsolution 22
B2 Secondary school environmental resilience asset exploratory actor analysis results, validation sample,8-actor solution 23
B3 Final secondary school environmental assets model, main sample 24
B4 Correlations among secondary school environmental resilience assets, nal conrmatory actor analysismodel 25
B5 Elementary school environmental resilience asset exploratory actor analysis results, main sample, 4-actorsolution 25
B6 Elementary school environmental resilience asset exploratory actor analysis results, validation sample,4-actor solution 26
B7 Final elementary school environmental resilience assets model, main sample 27
B8 Secondary school internal resilience asset exploratory actor analysis results, main sample, 4-actormodel 28
B9 Secondary school internal resilience asset exploratory actor analysis results, validation sample, 4-actormodel 29
B10 Final secondary school internal resilience assets model, main sample 30
B11 Elementary school internal resilience asset exploratory actor analysis results, main sample, 2-actormodel 30
B12 Elementary school internal resilience asset exploratory actor analysis results, validation sample, 2-actormodel 31
B13 Final elementary school internal resilience asset model, main sample 31
B14 Secondary school internal consistency reliability coecients by demographic subgroup 32
B15 Elementary school internal consistency reliability coecients by gender 32
B16 est-retest reliability o secondary school environmental resilience asset constructs and items 33
B17 est-retest reliability o secondary school internal resilience asset constructs and items 34
B18 est-retest reliability o elementary school resilience asset constructs and items 35
B19 Secondary school subscale means by demographic subgroup 36
B20 Elementary school subscale means by gender 37
B21 Correlations between secondary school environmental resilience assets and criterion variables 38
B22 Correlations between secondary school internal resilience assets and criterion variables 39
8/14/2019 description: tags: REL 2007034 body
8/22
B23 Correlations between elementary school resilience assets and criterion variables 40
B24 Current and recommended measures o environmental resilience assets among secondary schoolstudents 41
B25 Current and recommended measures o internal resilience assets among secondary school students 42
B26 Current and recommended measures o environmental resilience assets among elementary schoolstudents 43
B27 Current and recommended measurement o internal resilience assets among elementary schoolstudents 43
C1 Secondary school environmental resilience assets, total analytic sample, goodness-o-t inormation orexploratory actor analysis models 44
C2 Secondary school environmental resilience asset, total analytic sample, goodness-o-t inormation orconrmatory actor analysis models 46
C3 Measurement intercept dierences or environmental resilience assets, secondary school sample 47
C4 Elementary school environmental resilience assets, total analytic sample, goodness-o-t inormation orexploratory actor analysis models 48
C5 Elementary school environmental resilience asset, total analytic sample, goodness-o-t inormation orconrmatory actor analysis models 49
C6 Gender measurement intercept dierences or environmental resilience assets, elementary school sample 49
C7 Secondary school internal resilience assets, total analytic sample, goodness-o-t inormation or exploratoryactor analysis models 50
C8 Secondary school internal assets, total analytic sample, goodness-o-t inormation or conrmatory actor
analysis models 51C9 Measurement intercept dierences or internal resilience assets, secondary school sample 51
C10 Elementary school internal resilience assets, total analytic sample, goodness-o-t inormation orconrmatory actor analysis models 52
8/14/2019 description: tags: REL 2007034 body
9/22
Why thiS Study? 1
Ths tsmmzs
s m st thshmtts th s thmtm, kmt th HthKs S. Th
st ms tm sssssmt sh s thtts sh th shmtt mtms.
WHy THiS STudy?
As improvements to curriculum and instruction
raise academic standards, researchers are look-
ing more and more at what actors account or
the varied inuence o these improvements. Mosthave ocused on risk actors or academic ailure,
such as poverty or racial and cultural minority
status. But researchers are beginning to look at the
other side o r iskresilienceand have identi-
ed several traits common to resilient youth that
enable the youth to overcome barriers to academic
success. Tere is little research, however, on how
to measure these traits within the general student
population and how to determine the role o the
school environment in promoting these traits.
Te Healthy Kids Survey (HKS) is one o the ew
large-scale surveys to assess both risk and resil-
ience. Te surveys resilience and youth develop-
ment module (RYDM) is based on the premise
that youth who experience high levels o environ-
mental assets in three areashigh expectations
rom adults, caring relationships with adults, and
opportunities or meaningul participation
will develop the resilience traits, the connection
to school, and motivation to learn that lead to
positive academic, social, and health outcomes(Constantine, Benard, & Diaz, 1999).
Te resilience and youth development module
which has both elementary and secondary school
versionswas designed as an epidemiological
surveillance tool to track aggregate levels o pro-
tective actors. In Caliornia an average o about
600,000 students take the Healthy Kids Survey
and a part o the resilience and youth development
module every year. School districts and schools
use the resulting prevalence and trend data toguide programmatic decisionmaking. With such
widespread administration, school districts and
independent evaluators are increasingly using the
survey data to evaluate local programs by examin-
ing student-level changes over time. Capitalizing
on the mandated administration o a standard
instrument or local evaluation has the benet
o reducing the survey burden or students and
8/14/2019 description: tags: REL 2007034 body
10/22
2 meaSuring reSilience and youth development: the pSychometric propertieS o the healthy KidS Survey
provides comparable outcome data across dierent
program evaluations.
Widespread use o the module or research and
local evaluation may be premature, however.
Te psychometric properties o specic scalesassessed by the elementary school module have
yet to be established. And the secondary school
module has not been validated since 2000, when
the instrument was rst tested in the eld. Te
instrument has since been modied several times,
making validation o the current secondary school
resilience and youth development module neces-
sary. In addition, measurement equivalence across
racial and ethnic groups, males and emales, and
dierent grades has never been systematically
examined. Te stakes are thus high to ensure thatall parts o the module are valid and reliable.
o guide urther improvements o this important
assessment tool, Regional Educational Labora-
tory West conducted psychometric analyses o the
properties o the resilience and youth development
module, using a large set o recent survey data.1
Tis report describes the results o these analyses,
makes recommendations on the proper use o the
module, and suggests modications to improve
the instrument.
For the secondary school module, the results
are consistent with the instruments current use
as an epidemiological tool and with its concep-
tual oundation. It provides comprehensive and
balanced coverage o eight environmental resil-
ience assets and our internal resilience assets;2
its subscales exhibit good internal consistency
and are associated with student risk actors in
expected ways. And i certain items are dropped,
the module also demonstratesmeasurement equivalence across
racial/ethnic groups, males and
emales, and grades. Te second-
ary school RYDM scales exhibit
low test-retest reliability, however,
which suggests that the module
is not well suited or examining
student-level changes over time.
Te instrument was not designed to examine
individual dierences across students and should
not be used this way. Moreover, two o the six
internal assets that the secondary school module
was designed to measurecooperation and goals/
aspirationscould not be assessed validly. Severalmeasures would benet i additional items were
included in derived scales to increase domain
coverage.
Te elementary school module was designed to
assess seven environmental resilience assets and
three internal resilience assets, but it can reli-
ably assess only two environmental assets and
one internal asset. Most o the scales measured
by the elementary school instrument have poor
psychometric properties. Te elementary schoolinstrument should thus be modied considerably
to make it suitable or research.
developing a riSK and reSilience
aSSeSSMenT Tool
Te Healthy Kids Survey is a comprehensive health
risk and resilience data collection system that
relies on student sel-reporting. Te surveys core
module tracks health risks and problem behaviorsthat are signicant barriers to learning among
students. Te resilience and youth development
module assesses individual and environmental
assets associated with positive youth development
and school success. Tis section provides a brie
background on how the survey and the resilience
and youth development module were developed
and are now used in Caliornia.
The Healthy Kids Surveyassessing
risk and protective factors
Te Healthy Kids Survey is the largest eort in the
nation to require school districts to assess student
resilience and risk behaviors (box 1). Te Calior-
nia Department o Education requires all school
districts with ederal itle IV unding or with state
obacco Use Prevention and Education grants to
administer the survey every two yearsthe case
h s th
mt m
sssss
mt ssts
sst wth st
th mt
sh sss
8/14/2019 description: tags: REL 2007034 body
11/22
developing a riSK and reSilience aSSeSSment tool 3
or 85 percent o Caliornia school districts. Inmandating the survey, the Caliornia Department
o Education aims to promote accountability and
data-driven decisionmaking and to improve health
and prevention programs in schools.
Te survey was developed in 1997 by WestEds
Health and Human Development Program in col-
laboration with Duerr Evaluation Resources and
an advisory committee o researchers, teachers,
prevention and health program practitioners,
and public agency representatives. Te CaliorniaDepartment o Education unded the develop-
ment o the survey in response to ederal require-
ments that schools implement the Principles o
Eectivenessto collect and use data to assess
student needs, justiy program unding, guide
program development, and monitor progress in
achieving program goals. Te immediate impetus
or mandating the biennial administration o the
survey, however, was meeting the requirements othe No Child Lef Behind Act (itle IVSae and
Drug-Free Schools and Communities Act).
Te Healthy Kids Survey consists o a general core
module, the resilience and youth development
module, and our optional modules on specic
risk behaviors. It can be customized to meet local
needs:
Te required core module assesses demo-
graphic inormation and health risks relat-ing to school violence, harassment, physical
health, mental health, school-related behavior
(such as truancy), and alcohol, tobacco, and
other drug use.
Te resilience and youth development module
assesses environmental actors (environmental
assets) and individual traits (internal assets)
Box 1
Specifcations o the HealthyKids Survey
Mandate
Mandated (since all 2003) by the
Caliornia Department o Educa-
tion or compliance with No Child
Lef Behind and state obacco Use
Prevention and Education (UPE)
grants
Survey type
Comprehensive health r isk and
resilience survey
Student sel-report
Anonymous, voluntary,
condential
Modular secondary school
instrument; single elementary
school version
Grade levels
Grades 5, 7, 9, 11, and students in
continuation schools
Sampling
Representative district sample;
school-level surveys optional
Required modules
(secondary school)
Core (required)A.
Resilience and youth develop-B.
ment (school and community
asset scales required)
Optional modules
(secondary school)
Resilience and youth develop-B.
ment (home, peer, and internal
asset scales)
Saety (violence and suicide)C.
and alcohol and other drug
use
obaccoD.
Physical healthE.
Sexual behavior (pregnancy andF.
HIV/AIDS risk)
Custom module (or addingG.
questions)
Sources
Items based on the Caliornia Student
Survey, Youth Risk Behavior Survey,
and Caliornia Student obacco Use
and Evaluation Survey
Requirements
Biennial administration
Module A and school & commu-
nity asset scales in module B
Module D by state UPE grantees
Written parental consent; passive
consent optional since all 2004
Representative district samples
Administration
By school, ollowing detailed
instructions
Processing and reporting by
WestEds Health & Human De-
velopment Program
Product
Local reports and aggregated state
database
8/14/2019 description: tags: REL 2007034 body
12/22
4 meaSuring reSilience and youth development: the pSychometric propertieS o the healthy KidS Survey
associated with academic perormance, posi-
tive youth development, and protection rom
risky behaviors. Te Caliornia Department o
Education mandates that the sections on school
and community assets be administered to all
students who take the Healthy Kids Survey.
Four optional, topical modules (and one
customizable module) collect urther detail
on subjects covered by the core module, such
as violence and alcohol and other drug use
(module C); tobacco use and tobacco educa-
tion (module D); physical activity and diet
(module E); and sexual behavior, pregnancy,
and HIV risk (module F).
A custom module that allows schools to incor-
porate their own items.
Te survey was designed as a district surveillance
tool to provide prevalence estimates representative
o students in the school districts that administer the
survey rather than o students in the state as a whole.
It was not designed to evaluate student-level changes
over time or individual dierences across students.
Te Caliornia Department o Education requires
that districts administer the survey to 900 randomly
selected students rom each targeted grade (5, 7, 9,and 11). In districts with ewer than 900 students per
grade (the case or 85 percent o Caliornia districts),
all students in the targeted grades are surveyed. I
a district has more than 10 schools per grade, at
least 50 percent o schools are randomly sampled.
(Los Angeles Unied School District has dierent
requirements because o its size.)
WestEds Health and Human
Development Program provides
school districts administeringthe survey with technical as-
sistance and with a report on the
district-level data collected in each
module.
Although several adolescent
behavior surveys, such as the
Youth Risk Behavior Surveillance
System, assess student risk actors and problem
behavior, the Healthy Kids Surveys assessment o
student supports, strengths, and competencies sets
it apart. While some surveys incorporate protec-
tive actors, the resilience and youth development
module is one o the ew assessments that speci-cally addresses this dimension and does so with a
strong theoretical oundation.
The resilience and youth development module
assessing the other side of risk
Secondary school module.In early 1998 the HKS
Advisory Committee asked WestEd to develop a
survey module to assess middle and high school
student strengths, competencies, and positive so-
cial and health attitudes, eeling that the HKS coremodule did not give practitioners enough inorma-
tion about the actors behind positive development
and school success (Constantine et al., 1999).
WestEd ormed a Resilience Assessment Expert
Panel to develop and validate a new survey module
on youth resilience. Te assessment needed to be
brie enough to be widely administered along with
the HKS core module; have a strong theoretical
oundation; demonstrate reliability, validity, and
cultural and developmental appropriateness whenadministered in Caliornia school settings; and
provide a comprehensive, research-based assess-
ment o environmental actors (environmental
assets) and resilience traits (internal assets).
Environmental assets reer to meaningul and pro-
social bonding to community, school, amily, and
peers. Internal assets are personal resilience traits,
such as sel-ecacy and problem-solving skills
(Benard, 1991, 1995, 2004).
Failing to nd a survey that met its theoreti-cal and psychometric criteria, the panel built
on research on resilience and healthy human
development systemsparticularly the work o
Benard (1991, 1995, 2004)to develop a theoreti-
cal ramework that describes resilience actors
and their interrelationships (gure 1). Te result-
ing module or secondary school students was
designed to measure 11 environmental assets,
t s
ht mt ts tht
shmt
t, th t
sh t
tht mwk
ht ss s
ts th
ttshs
8/14/2019 description: tags: REL 2007034 body
13/22
developing a riSK and reSilience aSSeSSment tool 5
asking students their perception o adult high
expectations, their perceptions o caring rela-
tionships with adults, and their opportunities
or meaningul participation in school, home,
and community environments. Te module alsoassesses caring relationships and high expecta-
tions in the peer domain. Tese external sup-
ports promote positive outcomes, discouraging
risky behavior and stimulating academic success
(Benard, 2004; Constantine et a l., 1999; Hawkins,
Catalano, & Miller, 1992; Masten & Coatsworth,
1998; Resnick et al., 2000; Rutter, 1987; Werner &
Smith, 1982, 1992).
Internal resilience assetsthe personal strengths
o a resilient childinclude social competence,problem solving, autonomy, and sense o pur-
pose, which can each be broken down urther
(Benard, 1991, 2004). Socia l competence, or ex-
ample, entails social communication skills, em-
pathy and caring, and the ability to elicit positive
responses rom others (responsiveness) (Benard,
2004; Masten, 2001). Problem solving involves
planning, exibility, and resourceulness;
autonomy entai ls sel-ecacy, sel-awareness,
and mindulness; and sense o purpose in-
cludes goal direction, achievement motivation,
optimism, and hope (Benard, 2004). Internal
resilience assets develop both naturally and inresponse to environmental resilience assets. Te
resilience and youth development module was
designed to measure six internal assets: empathy,
problem solving, sel-ecacy, sel-awareness,
cooperation and communication, and goals and
aspirations.
A pool o 128 potential items was piloted in one
middle and one high school in all 1998. Re-
searchers, classroom teachers, and other school
practitioners helped select and modiy itemsrom the pool and revise the ormat and instruc-
tions. Te rst eld test o the resilience and
youth development module, with 92 resilience
items, was administered to 1,000 high school
students in three school districts in winter 1999.
Cognitive processing interviews with students
were also conducted to nd out students inter-
pretation o the items. Based on analysis o the
Improved
health,
social, and
academic
outcomes
Internal resilience assets
Environmental resilience assets Youth needs
School
Home
Community
Peers
igure 1
ct m th s th mt m
8/14/2019 description: tags: REL 2007034 body
14/22
6 meaSuring reSilience and youth development: the pSychometric propertieS o the healthy KidS Survey
taBle 1
itms th s sh s th mt m stt, 2006/07
cs i ds
Environmental resilience assets
Sh ssts
c ss s
SchlCare r6
r8
r10
a s, s s w. . .
s b .
s w i .
ss w i s s.
h s s
SchlHigh r7
r9
r11
a s, s s w. . .
s w i jb.
ws ws bs.
bs i w b sss.
m s
SchlPart r12
r13
r14
a s. . .
i s s.
i s k ss s s.
i s k .
Hm sstsc ss
HomeCare r49
r51
r53
i , s s . . .
w s s swk.
w ks w b bs.
w ss w i s s.
h s
HomeHigh r48
r50
r52
i , s s . . .
w s w s.
w bs i w b sss.
w ws ws bs.
m
HomePart r54
r55
r56
a . . .
i s s w s s.
i s k .
i k ss w .
cmmt ssts
c ss
ComCare r15
r17
r20
os s, s . . .
w s b .
w s w i s b s.
w i s.
h s
ComHigh r16
r18
r19
os s, s . . .
w s w i jb.
w bs i w b sss.
w ws ws bs.
m
ComPart r21
r22
r23
os s, i s s. . .
i bs, ss s, /, s.
i s, , , ss bb.
i .
p ssts
c ss w s
PeerCare r42
r43
r44
i b w . . .
w s b .
w ks w b bs.
w s w i .
p-s s
PeerHigh r45
r46
r47
m s. . .
b.
w s .
w s.
8/14/2019 description: tags: REL 2007034 body
15/22
developing a riSK and reSilience aSSeSSment tool 7
cognitive interview data, requency distributions,and estimated Cronbachs alpha coecients, the
number o resilience items was reduced rom
92 to 51 (table 1). In 2001 the resilience instru-
ment was modied again, based on the results o
grade-, gender-, and race/ethnic-specic explor-
atory actor analyses o data collected during
the 1999/2000 academic year. Te constructed
resilience scales based on the 1999/2000 eld test
data orm the basis o the current RYDM reports
provided to school districts, even though the
module has since been modied urther.
Since 2003 all districts administering the Healthy
Kids Survey must also administer the school and
community asset parts o the module.3 Tirty-ve
percent o districts choose to administer the ull
resilience and youth development module, reect-
ing widespread interest in assessing resilience.
WestEd provides districts with the data or each
scale and a report on the meaning and use o thedataand on how schools can create supportive
learning environments that promote school con-
nectedness and achievement. WestEd also pro-
vides state-level data to researchers and evaluators
who apply or it.4
Elementary school module. Pools o resilience
items were not independently developed or the
elementary school module. Tey were selected
rom the secondary school module afer ocus
groups with elementary school students. Initially,the elementary school module used the same
constructs as the secondary school module, but
with two items per construct instead o three.
Analysis o the 1999 eld test data and cognitive
processing interviews with students suggested
item deletions and changes in item wordings and
response options. Te nal version has 21 items
(table 2).
cs i ds
Internal resilience assets
c
Coop r31
r36
r37
hw s ss b s?
i wk w s w s s .
i j wk w ss .
i s s w s w.
S-
SelEf r29
r30
r32
hw s ss b s?
i wk bs.
i s s i .
t s i w.
e
Empathy r33
r34
r38
hw s ss b s?
i b w s s s .
i s w .
i s w k.
pb-s
ProbSolv r35
r27
r28
hw s ss b s?
W i i f s k w.
i kw w w b.
i wk bs b k w b .S-wss
SelAware r39
r40
r41
hw s ss b s?
t s s .
i s s s.
i s w i w i .
gs ss
Goals r24
r25
r26
hw s ss b s?
i s s .
i s.
i s s s.
Note: Possible responses include (1) not at all true, (2) a little true, (3) pretty much true, (4) very much true.
8/14/2019 description: tags: REL 2007034 body
16/22
8 meaSuring reSilience and youth development: the pSychometric propertieS o the healthy KidS Survey
evaluaTing THe pSycHoMeTric
properTieS o THe reSilience and
youTH developMenT Module
o better understand and improve the psychomet-
ric properties o the resilience and youth develop-
ment module, this report analyzes local HKS data
processed between 1998 and spring 2005, askingthe ollowing questions:
How should school districts and local evalu-
ators best use the module? Should the instru-
ment be used exclusively to assess prevalence
o environmental and internal assets or
should it also be used to assess student-level
changes across time?
What are the psychometric properties o
specic scales assessed by the secondary and
elementary school resilience and youth devel-
opment modules (including the dimensional-
ity o scales, scale reliability, and construct
validity)?
Does the module exhibit measurement
equivalence across racial and ethnic groups?
In other words, is it culturally appropriate
or dierent racial and ethnic groups? Does it
exhibit measurement equivalence or males
and emales? Across dierent grades?
What modications should be made to im-
prove the module?
taBle 2
emt sh s th mt m tms stt, 2006/07
cs i ds
Environmental resilience assets
Sh ssts
c ss s
SchlCare
10
13
d s w-s s b ?
d s w-s s s w s s?
h s s
SchlHigh
11
14
d s w-s s w jb?
d s w-s s b jb?
m s
SchlPart
9
15
d k ss s s s s?
d s b s?
Hm ssts
c ss
HomeCare
52
55
ds s w- b swk?
ds s w- s w
s s?
h s
HomeHigh
53
54
ds s w- b jb?
ds s w- w bs?
m
HomePart
56
57
d ?
d k s s s ?
p ssts
h s w s
PeerHigh
50
51
d bs s b?
d bs s ?
Internal resilience assets
e
Empathy
37
38
d s w ?
d b w s s s ?
pb-s
ProbSolv
39
40
d kw w w b?
d wk bs b k w b ?
gs ss
Goals
41
4216
d bs?
d s s ?d s s s?
Note: Possible responses include (1) no, never, (2) yes, some of the time, (3) yes, most of the time, (4) yes, all of the time.
8/14/2019 description: tags: REL 2007034 body
17/22
evaluating the pSychometric propertieS o the reSilience and youth development module 9
Box 2
Data and analytic strategies
Te authors used the ollowing data
and analytic strategies to analyze
the psychometric properties o the
secondary and elementary school
resilience and youth development
modules.
Data
wo mutually exclusive analytic
samplesa main sample and a
validation samplewere drawn
rom an aggregate data le that
included all HKS data processed
between the spring 2003 and the
spring 2005 administrations o the
Healthy Kids Survey. For the second-
ary school analysis, separate samples
were drawn or each grade (7, 9, and
11), gender, and ethnicity (Chinese
American, Arican American, Mexi-
can American, and white European
American)with 500 respondents
randomly sampled per cell (12,000
total). Equal numbers were used or
each gender and ethnic group so that
models that do not adjust or gender
and/or ethnicity would not be a-
ected by gender/ethnic dierences in
the sample.
For the elementary school analysis,
random samples o 1,000 males and
1,000 emales (2,000 total) were
drawn rom the aggregated HKS data
le. Tus, or the elementary school
resilience and youth development
module, only gender dierences in
measurement structure were exam-
ined. Respondents with missing data
on more than hal the resilience items
were excluded rom the analysis. For
estimating models with missing data,
maximum likelihood estimation with
missing at random (MAR) assump-
tions were used, which assumes that
values are missing at random con-
ditional on the other observed items
in the data (Little & Rubin, 2002;
Muthn & Muthn, 2006).
Statewide data was supplemented
with two sets o HKS data originally
collected or local evaluation. Data
collected in 2006 rom a large urban
school district in Southern Caliornia
were used to describe the temporal
stability o the derived scales (test-
retest reliability). Te elementary
school Healthy Kids Survey and the
secondary school core module and re-
silience and youth development mod-
ule were administered two times in
two weeks to 132 fh-grade students
and 90 ninth-grade students. Data
collected in 2004/05 rom students in
a large county in Southern Caliornia
were used to examine the relation-
ship between the RYDM constructs
and standardized test scores.
Exploratory and confrmatory
actor analyses
Analyses were conducted to test em-
pirically whether the actor structure
o the resilience instrument is con-
sistent with current usage and with
its underlying conceptual model. For
each sample and subsample (grade,
gender, ethnicity), the measurement
structure o the resilience instrument
was established by tting a series o
exploratory and conrmatory actor
analysis models. Exploratory actor
analysis (EFA) models were estimated
to determine roughly the number o
actors underlying the data and the
measurement structure o the latent
actors. A combination o criteria was
used to determine the number o ac-
tors to retain in the EFAs, including
t indices, scree plots, the number o
eigenvalues greater than 1, concep-
tual clarity, and simplicity. Models
with the ewest possible actors and
models with no cross-loadings were
avored over more complex models.
Te results o the exploratory actor
analysis models were then used as a
starting point or a series o nested
conrmatory actor analysis (CFA)
models. Measures o model t, cor-
relations among the latent constructs
(actors), and actor-loading patterns
were used to make decisions about
models. Tis process was replicated
or each grade, gender, and ethnic
group, and or the main sample and
the validation sample.
o derive estimates or the EFA and
CFA models, Muthn and Muthns
(2006)Mplus statistical modeling
program was used. Because all the
items used to measure resilience
assets are ordinal, Muthns (1984)
approach to exploratory and conr-
matory actor analysis with ordinal
indicators was used.
Confrmatory actor analysis
models with covariates
Measurement equivalence across de-
mographic subgroups was examined
by estimating conrmatory actor
analysis models with covariates.
MIMIC modelingmultiple indica-
tor, multiple cause structural equa-
tion modelswas used to test or
dierential item unctioning across
school grade, gender, and ethnic-
ity. An applied strategy was used to
(continued)
8/14/2019 description: tags: REL 2007034 body
18/22
10 meaSuring reSilience and youth development: the pSychometric propertieS o the healthy KidS Survey
Tis report nds that both the secondary school
and elementary school modules are used pri-
marily to report aggregate data on prevalence
and district-level changes across time. Although
several modications should be made, the RYDM
scales are generally consistent with current use o
the instruments and with the conceptual ounda-
tion o the module. (See box 2 and appendixes Aand B or a discussion o the analytic strategy and
the results o the analysis.)
Results of the analysis of the secondary school module
Te secondary school module is a short instru-
ment (51 items) suitable or widespread adminis-
tration. It provides comprehensive and balanced
coverage o both environmental (eight dimen-
sions) and internal (our dimensions) resilience
assets.5 Its subscales exhibit good internal consis-
tency and are associated with student risk actors
in expected ways. I certain items are dropped,
the module also demonstrates measurement
equivalence across racial/ethnic groups, males and
emales, and grades.
Te secondary school instrument is appropriate
as an epidemiological tool, but is not well suited
or evaluating student-level changes over time or
individual dierences across students. Te instru-
ment exhibits low test-retest reliability, suggesting
that the RYDM constructs are temporally specic.
Estimates o student-level changes across time are
ascertain whether group dierences
in measurement intercepts have
implications or evaluation research.Recommendations or item changes
are made only when the measure-
ment intercepts are substantively di-
erent across groups ( 0.20 standard
deviations) in both the main sample
and the validation sample.
Additional reliability and
validity analyses
Internal consistency estimates o
reliability o the derived scales werecalculated using Cronbachs alpha or
each grade, gender, and ethnic group
in both the main sample and the
validation sample. Nunnalys (1978)
criterion o 0.70 was used as the cuto
or determining acceptable internal
consistency reliability or the second-
ary school survey. Because o the no-
toriously low internal consistency evi-
dent in surveys o elementary school
students, this criterion was relaxedslightly to 0.60 or the elementary
school module. o examine test-retest
reliability, RYDM survey data col-
lected rom a small sample o fh andninth graders who took the resilience
and youth development module twice
in two weeks was used.
Dierences in resilience scale scores
across the demographic subgroups
were also examined. o make demo-
graphic dierences in the resilience
scales more interpretable, eect sizes
were calculated to represent the mag-
nitude o such dierences (Cohen,1988). With two groups (male/emale),
the dierence in scale means between
each group was divided by the pooled
standard deviation (Cohens d). Tus
the standardized dierence represents
the dierence between each group in
standard deviation units. With more
than two groups (race/ethnicity),
the standardized dierences were
represented by multiplying Cohens
by 2which is roughly equivalent tothe standardized dierence calculated
or two groups when the number
o observations in each cell is equal
(Cohen, 1988).
Construct validity was assessed by
examining the relationship o the
derived resilience scales to other theo-
retically related constructsinclud-
ing substance use, school violence,
school-related behavior, and stan-
dardized test scores. o examine these
relationships using a common metric,
correlations between resilience con-
structs and criterion variables romconrmatory actor analysis models
were estimated using the main and
validation samples. Latent constructs
represent continuous variables, while
the criterion variables are either
dichotomous or ordinal. Tus, polyse-
rial correlations are presented, which
represent the correlation between a
continuous variable and a dichoto-
mous or ordinal variable that reects
an underlying continuous variable(Bedrick & Breslin, 1996).
Box 2 (continued)
Data and analytic strategies
8/14/2019 description: tags: REL 2007034 body
19/22
evaluating the pSychometric propertieS o the reSilience and youth development module 11
likely to be imprecise because o the instability o
the resilience measures. Even with low student-
level stability, however, the module is valuable or
tracking school and district prevalence estimates
o resilience assets. Student-level errors in mea-
surement likely cancel each other out when thedata are aggregated at the school, district, and
state levels.
Te secondary school module contains eight in-
ternally consistent and valid measures o environ-
mental resilience assets:
Tree measures representing supportive rela-
tionships in the school, community, and home
environments. Tese supportive relationships
include both caring relationships with andhigh expectations messages rom adults. Only
the measure or supportive relationships in
the home environment, however, demon-
strates sucient test-retest reliability or use
in research.
Tree measures o meaningul participation
or involvement in relevant, engaging, and
interesting activities with opportunities or
responsibility and contribution in school, in
the community, and at home.
wo measures o environmental assets in the
context o peerscaring relationships and
high expectations (aliation with pro-social
peers).
Tat the scales or caring relationships and high
expectations in the school environment turn out
to measure the same actor is consistent with
knowledge that has emerged since the resilience
and youth development module was developed inthe late 1990s. In ocus groups conducted by HKS
sta, when students were asked what they consider
to be actions that reect that a teacher cares about
you, they most ofen mentioned that the adult
is a good listener, sets high standards, expects
responsibility rom the student, praises successes,
and encourages the student through setbacks.
Akey (2006) ound that supportive teachers and
clear, high expectations
or behavior are key to
developing both stu-
dent engagement and
perceived competence.
eachers whom studentssee as supportive and
who set clear expecta-
tions or behavior create
an atmosphere where
students eel in control
and condent about their ability to succeed in
school. Akeys ndings suggest that supportive
teacher relationships, high academic expectations,
and high-quality pedagogy combine to enhance
student engagement and academic competence,
which lead to higher achievement, consistent withthe RYDM conceptual ramework. Te school and
home supportive relationships measures, however,
exhibit better psychometric qualities than many
other instruments designed to measure similar
constructs.
Scores on our o the internal asset scalessel-
ecacy, empathy, problem solving, and sel-
awarenessare internally consistent and adequate
or general research purposes. But the RYDM
items designed to measure cooperation and goals/aspirations do not, however, provide valid assess-
ments o these constructs.
Although the consistency o the associations o
environmental and internal resilience assets to
other related constructssuch as substance use,
school violence, school-related behavior, and stan-
dardized test scoressuggests that the measures
demonstrate construct validity, the associations
are weak. Tus the constructs exhibit only moder-
ate construct validity.
Results of the analysis of the elementary school module
Te elementary school resilience and youth devel-
opment module uses 21 items to assess seven en-
vironmental assets and three internal assets. Reli-
ably assessing so many actors with so ew items
is dicult, however, especially with a student
Th s
sh stmt
s t s
m t
t s t w st
t stt-
hs tm
f
ss stts
8/14/2019 description: tags: REL 2007034 body
20/22
12 meaSuring reSilience and youth development: the pSychometric propertieS o the healthy KidS Survey
sel-report instrument. Not
surprisingly, the module reliably
assesses only two environmental
asset measures and one internal
asset measure, leaving consider-
able room or improvement.
Te elementary school module
measures meaningul participa-
tion, pro-social peers, and supportive relationships
in the school and home environments, but only the
school supportive relationships and home support-
ive relationships scales exhibit suciently high in-
ternal consistency or urther use. Only one reliable
internal resilience asset measure was detected or
elementary school studentsempathy. Te second
actor detected, goals/aspirations, was not reliableenough to be recommended or urther use. Te
third actor, problem solving, was not identied.
recoMMendaTionS
Tis report recommends that neither the second-
ary school nor the elementary school resilience
and youth development module be used to evaluate
student-level changes over time or individual di-
erences across students. Estimates o student-levelchanges across time are likely to be imprecise be-
cause o the instability o the resilience measures.
Other, longer, companion instruments should be
developed to assess student-level changes. Te
resilience and youth development module is still
useul as an epidemiological surveillance tool or
reporting aggregate district-level data, however.
Te ollowing sections provide recommendations
to drop or revise specic items in the module.
ables 3, 4, and 5 present the recommended mea-sures. (See appendix tables B24, B25, B26, and B27
or a side-by-side comparison o the current and
recommended measures.)
Secondary school environmental resilience assets
Recommendation 1Combine the caring relation-
ships and high expectations items. o maximize
construct validity and reduce redundancy across
scales, the caring relationships and high
expectations items should be combined to orm
one scale representing supportive relationships.
Caring relationships and high expectations are
indistinguishable as currently measured by themodule. Te new supportive relationships scale
should continue to be assessed separately or
school, community, and home environments.
Recommendation 2Drop Item R23 (I help other
people).Tis item should not be used to indicate
community meaningul participation because the
item unctions dierently, and thus has a dier-
ent meaning, or emales and Mexican American
youth. A new item that taps involvement in activi-
ties in the community should be developed.
Recommendation 3Drop Item R54 (I do un
things or go un places with my parents or other
adults). Te item is not developmentally appro-
priate or older adolescents because 11th graders
report substantially lower participation in such
activities or a given level o home meaningul
participation. Tis item distorts developmental
trends on the home meaningul participation scale
and should be dropped. A dierent item should be
developed to replace it.
Recommendation 4Drop item R45 (My riends
get into a lot o trouble). Because it is a biased in-
dicator o pro-social peers or emales and Chinese
American students, an alternative item should be
developed to measure this construct.
Secondary school internal resilience assets
Recommendation 5Drop the cooperation/com-
munication construct. wo o the items used tomeasure cooperation/communication measure
more than one construct: Items R36 (I enjoy
working together with other students my age)
and R37 (I stand up or mysel without put-
ting others down). Item R31 (I can work with
someone who has dierent opinions than mine)
should be moved to the sel-ecacy scale. Te
measurement models suggest that this item
h mt sh
m sssss
tw mt
sst mss
t sst ms,
s
m mmt
8/14/2019 description: tags: REL 2007034 body
21/22
recommendationS 13
measures sel-ecacy better than it does coopera-
tion and communication.
Recommendation 6Drop the goals and aspira-tions construct. wo o the three items used to
measure this constructR24 (Goals and plans
or the uture) and R26 (I plan to go to college or
some other school afer high school)unction
dierently across racial/ethnic groups.
Recommendation 7Drop item R27 (I know
where to go or help with a problem). As an
indicator o problem solving, this item should be
dropped because it unctions dierently or males
and emales. An alternative item should be devel-
oped to assess problem solving.
Elementary school environmental and internal assets
Recommendation 8Develop more elementary
resilience items. Te elementary school resilience
and youth development module tries to assess too
many actors with too ew items. Because having
an elementary school resilience assessment that
taBle 3
rmm mss mt s ssts m s sh stts
cs i
S s
a w s b .
a w s w i .
a w ss w i s . . .
a w s w i jb.
a w ws ws bs.
a w bs i w b sss.
S
i s s.
i s k ss s s.
i s k .
c s
a w s b .
a w s w i s b . . .
a w i s.
a w s w i jb.
a w bs i w b sss.a w ws ws bs.
c i bs, ss s, /, . . .
i k sss s, , . . .
h s
a w s s s wk.
a w ks w b bs.
a w ss w i s . . .
a w s w s.
a w bs i w b sss.
a w ws ws bs.
h i s k .
i k ss w .
p ss
a w s b .
a w ks w b bs.
a w s w i .
p-s sm s w s .
m s w s.
8/14/2019 description: tags: REL 2007034 body
22/22
14 meaSuring reSilience and youth development: the pSychometric propertieS o the healthy KidS Survey
is aligned with the secondary school module is
important, additional resilience items should be
developed or the elementary school survey. Each
o the elementary school RYDM scales demon-
strates inadequate domain coverage and marginalinternal consistency, at least one additional item
should be developed or each o the school sup-
portive relationships, home supportive relation-
ships, and empathy subscales. wo additional
items should be developed or the meaningul
participation at school and at home scales i it is
retained in the survey.
Recommendation 9Combine the caring rela-
tionships and high expectations items.As with
the secondary school module, the caring relation-ships and high expectations items should be
combined to orm one scale representing support-
ive relationships in both the school environment
and the home environment.
Recommendation 10Drop meaningul participa-
tion. Te meaningul participation scale should
either be dropped or redeveloped because o low
internal consistency. Moreover, item R15 (Do
you do things to be helpul at school?) shouldnot be used to indicate meaningul participation
because the item unctions dierently or males
and emales.
Recommendation 11Drop pro-social peers.Te
pro-social peers scale should be dropped because
one o the two items used to measure it unctions
dierently or males and emales. Perhaps items
rom other instruments that assess this construct
should be used instead.
Recommendation 12Drop goals and aspirations.
Te goals and aspirations scale should be dropped
or modied because o its low internal consistency.
Recommendation 13Develop a sel-efcacy
measure. Items should be developed to assess
sel-ecacy because this important construct is
currently not assessed.
taBle 4
rmm mss t s
ssts m s sh stts
cs i
S-
i wk w s w s
s .i wk bs.
i s s i .
t s i w.
e
i b w s s
s .
i s w
.
i s w
k.
pb s
W i i f s
k w.
i wk bs b k
w b .
S-wss
t s s .
i s s s.
i s w i w i .
taBle 5
rmm mss mt
t s ssts m
mt sh stts
cs i
Environmental resilience assets
S s
d s . . . s
b ?
ts . . . s w . . .
s s?
ts . . . w
jb?
ts . . . b
jb?
h s
p . . . b s wk?
p . . . s w
s s?
p . . . b
jb?
p . . . w
bs?
Internal resilience assets
e
d s w
?
d b w s s
s ?