SHIFTS IN BELIEFS USING DATA COMPARISON

FINDINGS OF SCHOOL COUNSELOR BELIEFS ABOUT

ASCA NATIONAL MODEL SCHOOL COUNSELING PROGRAM COMPONENTS USING THE SCPCS

Trish Hatch, Ph.D. San Diego State University

Timothy Poynton, Ed.D. Suffolk University

OVERVIEW OF SESSION• Activity: What do you think? • Have school counselors’ beliefs shifted since the introduction of the

ASCA Model?• How have beliefs shifted for use of data, accountability and reporting

results?• Which areas are positive? • Has this affected school counselor behavior? • What work is still needed? • Sharing ideas for supporting school counselors who may not yet

believe in the value of data and accountability

SCPSC• Hatch and Chen-Hayes (2008) published article about school

counselors’ beliefs regarding the ASCA National Model School Counseling Program Components using the School Counselor Program Component Scale (SCPCS).

• 2002 Survey administered to over 1,200 ASCA members

• Established psychometric properties of the SCPCS

• Collected national baseline data on school counselor beliefs about certain program components in the ASCA National Model prior to its release in 2003.

ACTIVITY #1WHAT DO YOU PREDICT?

• To what extent do you think school counselor beliefs shifted on the SCPCS items since the introduction of the ASCA National Model?

• Note rankings from the 2002 survey are activities that align with the ASCA National Model.

• They are listed from most important to least important • What shifts do you predict are significant?

2009 SCPSC INSTRUMENT• The SCPCS was modified slightly for use in the 2009

study, based on feedback provided by Professional School Counseling reviewers

• The original 5 point scale, which ranged from 1 to 5 with named anchors at 1 (very important), 3 (moderately important) and 5 (not important) was modified to a 3 point scale using only the named anchors of the original scale.

USEABLE SURVEYS

• The initial email invitation yielded 285 usable surveys (46%)

• Remaining 332 participants completed the survey after the reminder email was sent.

• The online survey was a multi-page survey which contained a page with informed consent

SURVEY PARTICIPANTS• 617 participants • 27% (n = 165) elementary school counselors • 21% (n = 130) middle/junior high school counselors • 32% (n = 196) were high school counselors, 12% • (n = 72) worked at multiple levels• 4% (n = 27) school counseling program supervisors• Twenty-seven respondents (4%) did not answer this

question.

DEMOGRAPHICS• 83% Female (n = 512)• 15% Male (n = 95) 2% (n = 10) did not answer

• 86% percent of the respondents White (n = 512)• 4% identified as Black (n = 23)• 4% identified as Hispanic (n = 24)• 2% identified as Asian or Pacific Islander (n = 12)• 2% identified as multiracial/multiethnic (n = 13)• 1% identified as some other race/ethnicity (n = 6)• 10 respondents (2%) did not answer this question

MORE DEMOGRAPHICS• Forty-six percent of the respondents worked in suburban

settings (n = 286), 28% in rural settings (n = 171)• 24% in urban settings (n = 147)• 13 respondents (2%) did not answer this question • Average amount of experience working as school

counselor reported by respondents was 8.1 years (SD = 7.1).

Q #1 FACTOR ANALYSIS

• To assess the possible impact of changing the scale on the factor structure, reliability, and validity of the SCPCS, several analyses were performed in a manner identical to those reported by Hatch and Chen-Hayes (2008).

• Specifically, internal consistency estimates and a principal components analysis (PCA) were calculated in a manner identical to the initial SCPCS study.

FACTOR ANALYSIS

• Compared Factor Structures 2002-2009• 18 of 19 SCPSC items loaded on same factors • Previous item not retained in FA in 2002 (#19) loaded on

administrator support. • Eigenvalues of three of the four factors in the 2009 data

were greater than 1.00• The amount of variance each factor explained differed from

the 2002 data

SUMMARY RESULTS • Despite the differences between the two studies (scale change and

participation due to on-line administration), analyses revealed four distinct and internally consistent factors evidencing construct validity.

• The factor structure of the SCPCS remained intact in the second study.

• Internal consistency estimates exceeded acceptable requirements for the scale and each of the four subscales.

• Thus, the four subscales were found to be consistent and reliable.

EIGENVALUES

• 9.94 (Use of Data for Accountability, 52.60%)• 1.35 (Use of Data for Program Planning, 7.10%)• 1.14 (Administrator Support, 5.99%)• .89 (Mission, Goals, and Competencies, 4.71%)• Majority of the variance in the 2002 data was explained by the

Use of Data for Program Planning factor (43.50%)• Majority of the variance in the 2009 data was explained by the

Use of Data for Accountability factor (52.60%).

INTERNAL CONSISTENCY (CRONBACH’S ALPHA)

• Use of Data for Program Planning = .87 (.82); • Use of Data for Accountability = .91 (.80); • Administrator Support = .85 (.78); • Mission, Goals, and Competencies = .84 (.86) • All subscales evidenced acceptable reliability

characteristics using the commonly accepted criterion provided by Nunnally (1978).

ANOVA (SUBSCALE MEANS)• Statistically significant, F (3, 606) = 27.78, p < .001, η2 = .12, a small effect size. • Post-hoc tests with a Bonferroni correction revealed differences among all possible

pairwise comparisons:• Mission, Goals, and Competencies (M = 1.32, SD = .41)• Use of Data for Program Planning (M =1.45 , SD = .46 )• Use of Data for Accountability (M = 1.37, SD = .46) • Subscales all significant at the p < .001 level. • Observed differences between the Administrator Support (M = 1.41, SD = .44) and

Use of Data for Program Planning subscales was significant (p = .033)• Difference between the Administrator Support and Use of Data for Accountability did

not reach statistical significance (p = .074).

Q #2 RANK ORDER • To facilitate analysis of changes in perceived importance of the

items over time, individual items were rank-ordered based on observed mean scores.

• The use of rank-ordered lists of the 2002 and 2009 data facilitates comparison between the two administrations in spite of the change in the scale

• Allows conclusions to be drawn about the perceived importance of each item relative to all other SCPCS items.

RANK ORDER (PERCEIVED IMPORTANCE)

• To assess the perceived importance of each individual item in a comprehensive school counseling program, descriptive statistics (mean and standard deviation) were calculated.

• The SCPCS item means and standard deviations are presented in rank order in Table 2, along with the rank of the item based on the 2002 data.

CHANGES IN RANKS OF BELIEFS OVER TIME

GREATEST RANK ORDER SHIFTS(TOWARD VERY IMPORTANT)

• 13th 6th use of data to measure the outcome results of the school counseling program

• 17th 10th using various data student data to identify gaps

• 10th 5th use of data to demonstrate the impact of the school counseling program on student success in school

DISCUSSION POSSIBLE EXPLANATIONS • Are you SURPRISED by the findings? • What do YOU think contributed to these findings?• If not what expected, what factors contributed to your

differing perspective?

DISCUSSION • Notable shift in Use of Data for Program Planning factor

accounting for the majority of the variance in 2002 to the Use of Data for Accountability factor accounting for the majority in 2009.

• Consistent with the 2002 findings, the Use of Data for Program Planning item “included five items related to using data to target interventions and identify program foci” (Hatch & Chen-Hayes, 2008, p. 38).

TOP RANKED: DEVELOPING GOALS FOR PROGRAM• Setting goals within the school counseling program is a

required component of the Recognized ASCA Model Program (RAMP) (ASCA, 2012a)

• SMART goals are included in ASCA’s Making Data Work (Young & Kaffenberger, 2009).

• Setting program goals has now been added as a new topic in the foundation section of the ASCA National Model (2012b).

SIX TOP RANKED ITEMS (2009)CONSISTENT WITH ASCA MODEL

1. Develop goals for the counseling program2. Write a mission statement or philosophy3. Utilize school-wide and student data to design new counseling

activities4. Identify specific student competencies to which the school counseling

program, curriculum or activities contribute or align5. Use data to demonstrate the impact of the school counseling program

on student success in school6. Use data to measure the outcome results of the school

counseling program

GREATEST POSITIVE RANK ORDER SHIFTS WHAT DOES IT MEAN?

• Have school counselors begun to prioritize the value, importance, and necessity of using data in their school counseling programs?

• Provides positive feedback for ASCA and national leaders in school counseling who have lamented the paucity of outcome research and the lack of school counselor engagement in data driven activities and accountability (it’s a start)

GREATEST POSITIVE RANK ORDER SHIFTS WHAT DOES IT MEAN?

• Two items moved up 7 RANKINGS! o Data to identify gaps o Data to measure impact

• Shift aligns with consistent requests for school counselors to do more than use data to design and prepare program activities .

• School counselors have been encouraged to improve efficiency and effectiveness and to promote the schools counselors’ value as worth its cost or resource

RANK ORDER FINDINGS ALIGN WITH NATIONAL REFORM EFFORTS • Aligns with work of Education Trust Transforming School

Counseling Initiative (TSCI) • Central tenets of ASCA National Model (Closing the Gap

Action Plans)• The College Board’s National Office for School Counselor

Advocacy (NOSCA), TSCI • National Association of College Admissions Counselors

(NACAC)

ASCA 3RD EDITION (161 PAGES) • Data mentioned 124 times• Gaps is mentioned 60 times. • Results (124 mentions)• Accountability (48 mentions) • ASCA Model incorporates language from ethical guidelines

(ASCA, 2010) stressing the responsibility of school counselors to use equity-based data to identify, address, and resolve attainment, achievement, and opportunity gaps.

pretty irrefutable

FUTURE RESEARCH

• School counselors who responded ASCA members who have received firm directive from the professional association to answer the question: “How are students are different as a result of what school counselors do?”

• Future research with a more diverse sample is needed to understand more about the beliefs of non-ASCA member school counselors regarding data for planning and accountability.

GREATEST NEGATIVE RANK ORDER SHIFTS

• 8th 14th Delineating non-school counselors related activities

• 12th 18th Adhering to non-school counseling activities

• Is there a relationship between school counselor beliefs and behaviors regarding accountability and non-school counselor activities

• Are school counselors who produce student and program results less likely to be asked to perform non-school counselor activities?

CONSULTING “MORE” WITH ADMIN. SHIFTED -4 FROM 3RD TO 7TH

• Are SC’s more competent and confident with regard to improving their school counseling programs and thus, believe there is less need to consult with administrators?

• Have they already begun to consult “more” regularly with administrators and thus, do not necessarily need to consult even “more”?

PRESENTING RESULTS STILL LOW

• Consistently low ranking on both surveys in relationship to other items (16th & 15th)

• Despite strong and consistent messages have been sent for school counselors to share their results with stakeholders (Hatch, 2008; Dimmitt, et al., 2007).

• Decisions on the employment of counselors can be based on the results they produce (Duarte & Hatch, in review; Fladager, 2012).

• Further research is needed to understand factors contributing to lack of perceived importance in relationship to other items.

LIMITATIONS

• Not longitudinal• Email to on-line not paper pencil• Problems with email:

o Did they ever get it? Sent to Junk?o Or not respond?

RESEARCH NEEDED ON…• 83% of the respondents in both surveys identified as female. • 2009 slightly more diverse (86% white compared to 92% in 2002). • Largest increase Hispanic/Latino: doubled 2% in 2002 to 4% in 2009. • African American respondents increased minimally from 3% to 4%.• Compared to College Board Study: 78% female, 10% African

American, and 15% percent Hispanic or Latino (Bruce & Bridgeland, 2012).

• Future research is encouraged to compare national school counselors’ demographics to membership in professional associations.

AND FINALLY…

• 3rd edition ASCA Model reflects current practice • School counselors and Common Core

• Future research needed to access shifts 2002 2009 2016

What do YOU think the NEXT a shifts will be?

SHARING IDEAS

• Share ideas for supporting school counselors who may not yet believe in the value of data and accountability

• Ideas to support their future growth in this area…

PRIMARY REFERENCE(EXTENSIVE LIST AVAILABLE UPON REQUEST)

• Hatch, T., & Chen-Hayes, S. F. (2008). School counselor beliefs about ASCA

National Model school counseling program components using the SCPCS.

Professional School Counseling, 12(1), 34-42.

• Hatch, T. & Poynton, T. & Perusse, R. (in review). Shifts in beliefs using data

Comparison: Findings of School Counselor Beliefs about ASCA National Model

School Counseling Program Components Using the SCPCS

THANK YOU

• Tim Poynton tpoynton@suffolk.edu • Trish Hatch trish@hatchingresults.com (until July 1, 2013)

• Contact Tim if you have questions (Thanks Tim!)