Do You See What I See? Teachers' and School Psychologists' Evaluations of Naturally Occurring

This article was downloaded by: [University of Arizona]On: 17 December 2014, At: 12:29Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number:1072954 Registered office: Mortimer House, 37-41 Mortimer Street,London W1T 3JH, UK

Journal of Educational andPsychological ConsultationPublication details, including instructionsfor authors and subscription information:http://www.tandfonline.com/loi/hepc20

Do You See What I See?Teachers' and SchoolPsychologists' Evaluationsof Naturally OccurringGeorge H. NoellPublished online: 08 Jun 2010.

To cite this article: George H. Noell (1999) Do You See What I See? Teachers'and School Psychologists' Evaluations of Naturally Occurring, Journal ofEducational and Psychological Consultation, 10:2, 107-128, DOI: 10.1207/s1532768xjepc1002_1

To link to this article: http://dx.doi.org/10.1207/s1532768xjepc1002_1

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy ofall the information (the “Content”) contained in the publicationson our platform. However, Taylor & Francis, our agents, and ourlicensors make no representations or warranties whatsoever as to theaccuracy, completeness, or suitability for any purpose of the Content.Any opinions and views expressed in this publication are the opinionsand views of the authors, and are not the views of or endorsed byTaylor & Francis. The accuracy of the Content should not be reliedupon and should be independently verified with primary sources ofinformation. Taylor and Francis shall not be liable for any losses,actions, claims, proceedings, demands, costs, expenses, damages,

http://www.tandfonline.com/loi/hepc20

http://www.tandfonline.com/action/showCitFormats?doi=10.1207/s1532768xjepc1002_1

http://www.tandfonline.com/action/showCitFormats?doi=10.1207/s1532768xjepc1002_1

http://dx.doi.org/10.1207/s1532768xjepc1002_1

and other liabilities whatsoever or howsoever caused arising directlyor indirectly in connection with, in relation to or arising out of the useof the Content.

This article may be used for research, teaching, and private studypurposes. Any substantial or systematic reproduction, redistribution,reselling, loan, sub-licensing, systematic supply, or distribution in anyform to anyone is expressly forbidden. Terms & Conditions of accessand use can be found at http://www.tandfonline.com/page/terms-and-conditions

Dow

nloa

ded

by [

Uni

vers

ity o

f A

rizo

na]

at 1

2:29

17

Dec

embe

r 20

14

http://www.tandfonline.com/page/terms-and-conditions

http://www.tandfonline.com/page/terms-and-conditions

JOURNAL OF EDUCATIONAL AND PSYCHOLOGICAL CONSULTATION, 10(2), 107-128 Copyright O 1999, Lawrence Erlbaum Associates, Inc.

Do You See What I See? Teachers' and School Psychologists' Evaluations of Naturally Occurring

Consultation Cases

George H. Noel1 and Kristin A. Gansle Louisiana State University

Randy Allison Heartland Area Education Agency 11, Johnston, Iowa

Teachers and school psychologists participating in routine school-based, problem-solving consultation were surveyed regarding their perceptions of the referral concern, intervention development process, intervention outcomes, and their time investment. Respondents were surveyed using parallel forms on matched cases to permit analysis of perceptual agreement. Respon- dents reported developing and implementing multicomponent interventions that were predominantly designed for elementary school boys. Referral concerns included a range of problems that were rated by respondents as relatively severe. Across most of the items, school psychologists and teachers ex- hibited relatively high levels of agreement. The person to whom both the consultant and consultee reported feeling most accountable for the effectiveness of the intervention was the classroom teacher (i.e., the consultee). Time investment estimates indicated that consulting school psychologists may have been more directly involved in implementing interventions than traditional definitions of consultation would indicate.

Correspondence should be addressed to George H. Noell, 236 Audubon Hall, Depart- ment of Psychology, Louisiana State University, Baton Rouge, LA 70803. E-mail: [email protected]

Dow

nloa

ded

by [

Uni

vers

ity o

f A

rizo

na]

at 1

2:29

17

Dec

embe

r 20

14

108 NOELL, GANSLE, ALLISON

An extensive literature exists advocating the use of consultation between classroom teachers and educational specialists as a vehicle for providing interventions to students. Supportive efficacy, acceptability, and implementation data can readily be located within published research reports. Literature reviews contributed to advocacy for school-based consultation by reporting that available studies contained primarily positive findings regarding the effectiveness of the consultation (Gresham & Kendell, 1987; Sherdian, Welch, & Orme, 1996; West & Idol, 1987). Consultation was also found to be an acceptable and preferred activity for many educators (Kratochwill, Elliott, & Rotto, 1995; Reschly & Wilson, 1995). Finally, de- scriptions of the successful systemic implementation of consultation services (e.g., Gutkin, Henning-Stout, & Piersel, 1988; Ikeda, Tilly, Stumme, Volmer, &Allison, 1996) indicate that consultation can be a practical way to meet student and educator needs.

One recurrent feature of the school-based consultation literature is the extent to which studies used informant reports as a dependent measure (Gresham & Kendell, 1987; Sherdian et al., 1996; Witt, Gresham, & Noell, 1996). Potential consultation participants were surveyed regarding their preferences, attitudes, attributions, and perceptions of consultation. School psychologists, typically conceptualized as potential consultants, indicated that they preferred an increased consulting role (Costenbader, Swartz, & Petrix, 1992; Reschly &Wilson, 1995) and attributed much of the success or failure of consultation to consultee behavior (Martin & Curtis, 1981; Smith & Lyon, 1986). Surveys of teachers indicated that they were most likely to seek consultation regarding aggressive behavior (Alderman & Gimpel, 1996), were frequently willing to consult regarding curricular adaptation (Wade, Welch, & Jensen, 1994), and viewed consultation positively after exposure to it (Gutkin, 1980). Across surveys, educators generally indicated that consultation was a desirable professional practice.

Interestingly, informant reports remain the most common dependent measure in school-based consultation studies in which participants engage in consultation as part of the study (Sheridan et al., 1996). The extensive collection of survey and ratings measures produced a considerable database regarding participants' perceptions of consultation. Drawing from reviews indicating a preponderance of positive findings and the frequent use of consultee reports as a dependent measure (Sheridan et al., 1996), we think it reasonable toconclude that consultees described school-based consultation as effective and satisfactory professional practice. However, concerns were raised regarding the reliability and validity of consultee reports as well as the epistemological limitations of those studies that do not collect measures of behavior (see Noel1 & Witt, 1996; Sheridan et al., 1996; Witt et al., 1996).

Dow

nloa

ded

by [

Uni

vers

ity o

f A

rizo

na]

at 1

2:29

17

Dec

embe

r 20

14

CONSULTANT-CONSULTEE AGREEMENT 109

Data exist indicating that consultee-teacher reports may not agree with direct measures of behavior (e.g., Fuchs & Fuchs, 1989; Robbins & Gutkin, 1994). However, it is worth noting that consultees' verbal reports may be a valid measure of their perceptions or subjective appraisal without also being a valid measure of behavior.

The consultant's perceptions are another logical point of comparison for consultee-teacher reports in addition to direct measures of behavior. These comparisons are better characterized as agreement or interobserver reliability comparisons than as tests of validity. Similar perceptions of the referral concern, consultation process, and intervention may facilitate problem solving and intervention, whereas dissimilar views may under- mine the ability of the consultant and consultee to work together effectively. Erchul, Hughs, Meyers, Hickman, and Braden (1992) provided empirical support for the intuitively appealing assertion that shared perceptions of the consultation process are related to more favorable perceptions of consultation's outcomes. Erchul et al. found that the greater the extent to which the consultant and consultee shared similar views of the consultation process, the more favorably the consultee perceived the bene- ficialness of consultation, their own competence, consultant effectiveness, and client improvement.

Although there are a number of reasons to hypothesize that the development of a shared perception of the consultation process and the specific referral contributes to consultation's outcome (Bergan & Kratochwill, 1990; Erchul et al., 1992; Gutkin & Nemeth, 1997), the extent to which consultants and consultees develop similar perceptions during consultation has not been extensively investigated. The study of perceptual agreement in naturally occurring consultation requires collection of data from both the consultant and consultee matched to the same referral for cases that oc- cur as a routine part of problem solving in education. This study collected data from consulting school psychologists and classroom teachers regarding the same cases to examine the extent to which they shared similar perceptions of the referral concern, the intervention planning process, and intervention outcomes in those naturally occurring consultation cases.

Perceptions of the referral concern were examined because they are the focus of the request for consultation services. Perceptions of intervention planning were surveyed because this stage is a central consultation activity, and failure to communicate effectively at this point can contribute to subsequent implementation problems (Gresham, 1989). Perceptions of the services delivered to the student, intervention implementation, and effectiveness were surveyed. Data were also collected for an additional domain in which perceptual agreement was not the primary focus. Participants

Dow

nloa

ded

by [

Uni

vers

ity o

f A

rizo

na]

at 1

2:29

17

Dec

embe

r 20

14


were asked to estimate the amount of time they devoted to various consultation activities. These data were collected due to the importance of time costs to the implementation of consultation (Noel1 & Gresham, 1993), as well as the importance of implementation time as a factor affecting treatment acceptability (Elliott, 1988). Additionally, previous cost estimates (e.g., Bamett et al., 1997) indicated that the personnel costs of consultation may be substantial and higher than expected.

METHOD

Participants and Setting

Participants were certified school psychologists and the teachers with whom they consulted as part of their professional practice. All school psychologists were employed by a large intermediate educational agency in a Midwestem state that provided services to school districts with a total en- rollment in excess of 100,000 students. The intermediate education agency developed and implemented an agency-wide, problem-solving consultation model for the delivery of services to students, which was described previously (see Ikeda et al., 1996). The service delivery model used by the agency emphasized intervention and problem solving as primary professional practices for school psychologists. All school psychologists received workshop training provided by the intermediate education agency in curriculum-based measurement, progress monitoring, building assistance teams, and problem-solving consultation. Teachers were employed by lo- cal education agencies. Although training in each of these areas was available to teachers from the intermediate education agency, the extent to which they participated in training is unknown.

Survey Content

We developed the employed survey specifically for the purposes of this study (see Table 1). Item development was guided by consultation implementation and outcome issues raised in the literature (e.g., Bamett et al., 1997; Fuchs & Fuchs, 1989; Gresham, 1989; Gutkin, 1980; Noel1 &Gresham, 1993; Smith & Lyon, 1986), as well as by the intervention documentation procedures used by the agency employing the participating school psychologists. Psychologist and teacher forms of the survey were identical ex- cept for questions that asked about the other party. For example, psycholo-

Dow

nloa

ded

by [

Uni

vers

ity o

f A

rizo

na]

at 1

2:29

17

Dec

embe

r 20

14

TABLE 1 Survey Content

Variable Measurement Method -- - Demographic data

Years experience Student's grade Student's sex

Referral concern Primary concern Targeted most important concern Problem severity

Intervention development Intervention elements Respondent involvement in planning

Other's participant's involvement in planning"

Other's participant's knowledge for planning"

Parents involved in planning Parents involved in implementation Baseline data collected Outcome goals set

Intervention outcomes Behavior change

Treatment integrity

Satisfaction with outcome

To whom respondent felt accountable

Time estimates Preparation time Implementation time Progress monitor time Follow up meeting time

Fill in the blank Fill in the blank Forced choice: male,female

Forced choice (choices appear in Table 2) Forced choice: yes, no 7-point Likert scale ranging from 1 (mild)

to 7 (severe)

Forced choice (choices appear in Table 3) 7-point Likert scale ranging from 1 (not at

all) to 7 (great deal) 7-point Likert scale ranging from 1 (not at

all) to 7 (great deal) 7-point Likert scale ranging from I (not at

all) to 7 (great deal) Forced choice: yes, no Forced choice: yes, no Forced choice: yes, no Forced choice: yes, no

7-point Likert scale ranging from 1 (not at all) to 7 (Xeaf deal)

7-point Likert scale ranging from 1 (not at all) to 7 (exactly)

7-point Likert scale ranging from 1 (dissat- isfied) to 7 (satisfied)

Force choice (choices appear in Table 4)

Fill in the blank Fill in the blank Fill in the blank Fill in the blank

" On the psychologist form the other participant was specified as the teacher. In the case of the teacher form the other participant was the consultant.

Dow

nloa

ded

by [

Uni

vers

ity o

f A

rizo

na]

at 1

2:29

17

Dec

embe

r 20

14

712 NOELL, GANSLE, AL.LISON

TABLE 2 Identified Primary Concerns for Teachers and School Psychologists

Respondent

Concern

Psychologists Teachers

n % n %

Reading skills Other Attention to task Noncompliance Hitting students Talking out Work completion Writing skills Organizational skills Out of seat

Note. Respondents who reported multiple primary concerns were not included in this table.

gists were asked, "How involved was the teacher in planning the intervention?" On the teacher form the question asked, "How involved was the psychologist in planning the intervention?" The survey is available from George H. Noel1 on request.

Demographic information. Respondents were asked to indicate their years of experience, the target student's grade, and the student's sex.

Referral concern. Respondents were asked to identify the primary concern targeted by the intervention from a list of 11 choices (see Table 2). Poor math skills was a choice for primary concern but does not appear in Table 2 because none of the respondents selected it. They were also asked to indicate whether the concern targeted for intervention was actually the most important concern for that student. This question was asked to examine the extent to which consultation led to an intervention for each informant's primary concern for the student rather than a secondary concern. The severity of the problem prior to intervention was rated on a 7-point Likert scale ranging from 1 (mild) to 7 (severe).

Dow

nloa

ded

by [

Uni

vers

ity o

f A

rizo

na]

at 1

2:29

17

Dec

embe

r 20

14


Referral concerns on the survey were limited to student-related issues of learning and behavior. These concems were targeted because we and the supervisor of school psychologists for the intermediate education agency perceived them to be the dominant type of consultation occurring in practice. Additionally, data regarding student learning and behavioral concerns were perceived as most useful to the agency sponsoring the research. Finally, the desire to keep the instrument reasonably simple pre- cluded an exhaustive listing of all of the types of concems teachers and school psychologists encounter in practice.

Intervention development. Respondents were asked to identify all of the components included in the intervention from a list of 12 provided (see Table 3). Respondents were asked to rate how involved they were in planning the intervention, as well as how involved they thought the other survey participants were. Respondents also rated how knowledgeable they thought the other participants were in planning the intervention. These three items were rated on Likert scales ranging from 1 (not a t all) to 7 (a great deal). Respondents were also asked to report whether the parents were involved in intervention planning, whether

TABLE 3 Elements Included in Interventions Reported by School Psychologists and Teachers

Respondent

Psychologrsts Teachers

Agreement (%) n % n O h

Rewards School-home notes Other Modified instruction Self-monitoring Tutoring Modified assignments Negative consequences Lower level curriculum Peer helper Counseling Organizational checklists Modified tests

Dow

nloa

ded

by [

Uni

vers

ity o

f A

rizo

na]

at 1

2:29

17

Dec

embe

r 20

14


baseline data were collected, and whether outcome goals were set. Each of these three questions were presented as a yes or no forced choice.

Intervention implementation. Respondents were asked whether or not the student's parents were involved in implementing the intervention in a yes or no forced-choice format. They were also asked to rate the amount of behavior change, the degree to which they perceived that the intervention was carried out as planned, and their satisfaction with the intervention effects. These three items were rated on Likert scales ranging from 1 (no behavior change, implementation, or satisfaction) to 7 (high levels of behavior change, accurate implementation, or satisfaction). The final item asked participants to select all of the persons to whom they felt accountable for the success of the intervention (see Table 4).

Time estimates. Respondents were asked to estimate how much time they remembered devoting to intervention preparation, minutes per week for implementation, minutes per week for progress monitoring, and total minutes devoted to follow-up meetings. Respondents were also asked how long the intervention was implemented.

TABLE 4 Persons to Whom Consultation Participants Felt Accountable

Respondent

Person

Psychologists Teachers

Agreement (%) n % n %

Self Partner in consultation: The

consultant or consultee Student Parents Other agency staff Building Assistance Team Principal Other students Counselor

Dow

nloa

ded

by [

Uni

vers

ity o

f A

rizo

na]

at 1

2:29

17

Dec

embe

r 20

14


Procedures

All of the school psychologists at the intermediate agency were sent two survey pairs in the fall and again in the winter of the same academic year. A survey pair consisted of one psychologist form and one teacher form. The survey was accompanied by a letter from the supervisor of school psychologists explaining the purpose of the survey and procedures. The letter ex- plained that responses would remain anonymous and that the purpose of the survey was to collect data regarding participants' perceptions of ongo- ing problem-solving consultation services. The survey was also described in two meetings attended by the school psychologists. For the year, 172 survey pairs were distributed.

Psychologists were asked to complete the psychologist forms from the last two cases for which they had consulted with a teacher regarding an intervention for a student. They were also asked to request that the teachers with whom they consulted on these two cases complete the teacher form of the survey. Surveys contained no identifying information for teacher or psychologist but were coded so that teacher and psychologist forms could be paired and analyzed when they were complete.

Surveys were distributed with envelopes to be returned by interagency mail service. Psychologists were reminded to return their surveys and to prompt their teachers to do the same during routine monthly meetings that were unrelated to this study. A total of 74 complete sets (43% return rate) of surveys were returned and analyzed. A complete set included a completed survey from both the consultant (i.e., school psychologist) and the consultee (i.e., teacher). An additional 20 psychologist forms were returned with no corresponding teacher form. A matching psychologist survey was obtained for all teachers who returned the survey.

RESULTS

Demographics

Psychologists reported a mean of 5.8 years (SD = 5.09) experience, whereas teachers reported a mean of 12.3 years (SD = 8.5) experience. Interventions described by the surveys were designed for 64 elementary school students (Grades K-5 = 87%), 8 middle school students (Grades 6-8 = 11%), and 2 high school or "other" students (2%). Fifty-eight students (78%) were boys, and 16 students (22%) were girls.

Dow

nloa

ded

by [

Uni

vers

ity o

f A

rizo

na]

at 1

2:29

17

Dec

embe

r 20

14


Referral Concern

School psychologists and teachers identified the same primary concern in 82.4% of cases. In 17 cases a respondent violated the instructions and reported more than one primary concern. If the respondent who reported multiple concerns included the primary concern indicated by the other respondent for that case, the case was coded as an agreement. Excluding cases for which a respondent selected multiple concerns, the percentage agreement was 74.5%. These agreement estimates can be considered con- servative because they require an exact agreement. A case in which the teacher wrote in "following directions" in the "other" category was coded as a disagreement with the psychologist's selecting the response choice of "refusal to comply with teacher directions," despite being quite similar. The percentage of psychologists and teachers selecting each concern as the primary reason for referral is presented in Table 2. Sixty-seven psychologists (91%) and 71 teachers (96%) reported that the intervention targeted the most important concern. In 90% of the cases, psychologists and teachers agreed that the most important concern was targeted. The mean problem-severity rating for each respondent as well as the correlation between their ratings are presented in Table 5.

TABLE 5 Agreement Between Teachers' and Psychologists' Likert Ratings of Intervention Variables

Respondent

Correlation Psychologists Teachers Dljference

Varlabld r pb M SD M S D M SD t ~ " f

Problem severity . l l .36 5.6 1.1 6.1 1.1 0.55 1.5 3.2 ,002 70 Teacher

involvement in planning .40 < .0001 5.6 1.3 6.2 1.4 0.49 1.5 2.8 ,006 72

Psychologist involvement in planning .10 .39 5.9 1.3 6.1 1.4 0.24 1.8 1.2 .25 73

Behavior change .69 < ,0001 4.6 1.7 4.6 1.7 0.03 1.4 0.18 .86 68 Satisfaction .55 < ,0001 4.7 1.7 5.3 1.7 0.52 1.6 2.7 ,008 70 Treatment

integrity .34 ,004 5.1 1.4 5.7 1.4 0.60 1.6 3.1 .003 69

"Variables were rated by respondent on a 7-point Likert scale with higher numbers indicating a greater amount. b~robability values are reported for descriptive purposes only. Due to the number of correlations and t tests calculated, these are not intended as tests of statistical significance.

Dow

nloa

ded

by [

Uni

vers

ity o

f A

rizo

na]

at 1

2:29

17

Dec

embe

r 20

14


TABLE 6 Agreement Between Teachers' and Psychologists' Ratings of Dichotomous Intervention

Respondent

Variables Agreement (%) Psychologists (%) Teachers (%)

Baseline data collected 71.8 76 85 Goals set before intervention

began 62.0 Parents involved in planning 71.2 Parents involved in

implementation 75.0

Intervention Development

Data for teacher planning, psychologist planning, parent involvement in planning, baseline data collection, and goal setting are presented in Tables 5 and 6. The percentage of cases in which teachers and psychologists reported using each intervention element, as well as the percentage agreement between them, is reported in Table 3. Teachers reported that, on average, the intervention included 4.1 elements, whereas psychologists reported an average of 3.0 elements. All respondents reported using at least 1 of the elements specifically listed on the survey.

Intervention Implementation

The data for parental participation in implementation, treatment integrity, behavior change, and satisfaction are presented in Tables 5 and 6. The correlation between consultant and consultee ratings for behavior change and satisfaction indicated moderate to strong agreement for perception of intervention outcome. The correlation between respondent ratings of treatment integrity was less strong, and school psychologists perceived implementation as slightly poorer.

Table 4 presents the data regarding to whom teachers and school psychologists reported feeling accountable for the effectiveness of the intervention. Both the consultant and the consultee reported feeling most accountable to the consultee (i.e., the teacher). Teachers reported feeling accountable to the consultant in the majority (7O0/0) of cases but reported feeling accountable to any other specific person in less than half the cases. Other specific persons included the target student and his or her parents.

Dow

nloa

ded

by [

Uni

vers

ity o

f A

rizo

na]

at 1

2:29

17

Dec

embe

r 20

14


Psychologists reported feeling accountable to students in slightly more cases and accountable to parents at rates similar to teachers.

Time Estimates

Teachers' and psychologists' estimates of the amount of time they spent on various intervention activities from preparation to follow-up are presented in Table 7.

Perceptions Related to Implementation and Outcome Ratings

To explore the relation between respondents' perceptions of consultation outcomes, a series of stepwise, forward, multiple-regression analyses were conducted for ratings of treatment integrity, behavior change, and satisfaction with outcome. Analyses were conducted separately for psychologists and teachers. At each step the variable that accounted for the most variance entered the equation. The probability to enter was set at 5%, and the probability to remove was set at 10%. Due the exploratory nature of these analyses, alpha was set at .05 for each analysis.

We selected the same pool of variables for all analyses. These were the informants' responses for problem severity; whether the target behavior

TABLE 7 Time Invested in Intervention Activities Reported by Teachers and School Psychologists

Respondent

Psychologists Teachers Di'erence

Element M S D M S D M S D t P" df

Total min preparation 109 81 76 164 33.3 184 1.41 .I6 60 Total min per week im-

plementation 31 67 37 70 5.9 73 0.62 .54 58 Total min per week

progress monitoring 11 13 28 59 16.7 60 2.03 ,048 52 Total min for follow-up

meetings 74 72 69 78 4.5 67 0.50 ,619 54 Length of intervention

(weeks) 10.6 8.6 13.3 9.6 2.7 6.4 2.89 .006 47

"Probability values are presented for descriptive purposes only. Due to the large number of comparisons they should not be interpreted as hypothesis tests.

Dow

nloa

ded

by [

Uni

vers

ity o

f A

rizo

na]

at 1

2:29

17

Dec

embe

r 20

14


was the most important concern; their involvement; other party's involvement; other party's knowledge; time in preparing, carrying out, and monitoring intervention progress; amount of time in follow-up meetings; and length of intervention. Although a specific a priori model was not specified for the analysis, the variables were selected based on the treatment acceptability literature (Elliott, 1988) and the cost-benefit analysis literature (e.g., Noel1 & Gresham, 1993), both of which indicate the importance of the severity of the referral concern and the time expended to resolve the problem. The knowledge and participation-in-planning variables were included based on the traditional emphasis on shared work within consultation (Bergan & Kratochwill, 1990).

Psychologist ratings. For psychologists, none of the variables examined was significantly related to ratings of behavior change or satisfaction with the outcome. Three variables were related to the psychologists' rating of intervention implementation. These included the psychologists' perception of teachers' knowledge, teachers' involvement, and their own involvement. The more the psychologist was involved and the more knowledgeable the teacher, the greater the psychologists' rating of integrity. However, the higher the psychologist's ratings of teacher involvement in intervention planning, the lower the psychologist rated treatment integrity. Results of this analysis are presented in Table 8.

Teacher ratings. Summary statistics for analyses of teacher ratings of behavior change, treatment integrity, and satisfaction are presented in Ta- ble 8. Teachers' ratings of behavior change and teachers' satisfaction with intervention outcomes were both positively related to their perceptions of the psychologists' knowledge. Additionally, the amount of time required to implement the intervention was negatively related to teacher satisfaction. Teachers' ratings of treatment integrity were negatively related to the time they reported spending preparing the intervention and positively related to their perception of the severity of the problem.

DISCUSSION

Referral Concern

Interventions targeted a range of primary concerns, with reading skills and disruptive or noncompliant behaviors being common targets for intervention. These findings may be partially an artifact of the procedures of the

Dow

nloa

ded

by [

Uni

vers

ity o

f A

rizo

na]

at 1

2:29

17

Dec

embe

r 20

14


TABLE 8 Summary of Stepwise-Forward Multiple-Regression Analyses of Variables Related to

Perceptions of Intervention Implementation and Outcome

Dependerit Variable Independent Variable B S E B P

Psychologist Treatment integrity" Step 1: Teacher knowledge

Step 2: Teacher involvement Step 3: Psychologist involvement

Teacher Behavior changeb Step 1: Psychologist knowledge Treatment integrityL Step 1: Teacher minutes preparation

Step 2: Teacher severity of behavior satisfactiond Step 1: Teacher implementation time

Step 2: Psychologist knowledge - - - - - - p p p p p p p p p - - - - - -

"R' = .084 for Step 1, A R = .I57 for Step 2, A R' = .062 for Step 3 (p < .05 for each step). 'R' = ,209 for Step 1 (p < .05).'R2 = ,220 for Step 1, A R'= .I16 for Step 2 (p < .05 for each step) *RR' = ,178 for Step 1, A R' = ,135 for Step 2 (p < .05 for each step).

study in that participants were asked to describe their last two consultations leading to interventions for students and the referral concems listed on the survey were all child focused. However, these findings were generally consistent with previous studies examining reasons for referrals for psychoeducational assessment (Lloyd, Kauffman, Landrum, & Roe, 1991; Ysseldyke, Christenson, Pianta, & Algozzine, 1983). These findings were also consistent with previous surveys of teachers indicating that they are likely to seek consultation regarding both aggressive behaviors and in- structional concerns (Alderman & Gimpel, 1996; Wade et al., 1994).

Notes to us in the margins of the surveys from some respondents indicated that a number of them did not believe that a single primary concern could be specified. Rather, they conceptualized the consultation and intervention as targeting several concerns. Asking participants to identify all of the concems they were attempting to address and then asking them to rank these may be a more valid way of describing referral concerns in future investigations. Alternatively, identifying critical and secondary concerns may have more utility in describing consultation. It may also be of interest in future studies to examine the process of problem identification and prioritization.

The finding that both psychologists and teachers rated the problems about which they were consulting as relatively severe is of some concern. If teachers are waiting until problems are severe to seek assistance, this re- duces the potential for consultation and intervention to involve secondary

Dow

nloa

ded

by [

Uni

vers

ity o

f A

rizo

na]

at 1

2:29

17

Dec

embe

r 20

14


prevention (Zins & Erchul, 1995). A number of factors might contribute to teachers delaying initiating consultation until problems are relatively severe. Potential consultants may not be readily available, teachers may be confronted with sufficiently many concems that they only seek outside assistance for severe problems, and consultation and intervention may be sufficiently time consuming that teachers only perceive it as warranted for especially difficult problems. Although these data cannot answer the question of why teachers did not seek consultation for mild problems, they do highlight the importance of understanding the variables influencing teacher requests for assistance. Consultation logically appears to have its greatest chance for success if intervention occurs before the problem is a long-standing, disturbing pattern of behavior (Bergan & Kratochwill, 1990; Zins & Erchul, 1995).

The finding that teachers rated problems as being somewhat more severe than the school psychologists is not surprising given the context of school-based consultation in the intermediate educational agency where the study took place. As a general regularity of practice in this setting, consultations were typically initiated by teachers. The referring teacher found the behavior distressing enough to initiate consultation about it and was likely to have been exposed to the behavior problem for much of the school day. In contrast, consulting school psychologists were almost assured to have had less exposure to the students and to have spent most of their work day focused on students who were exhibiting exceptional problems. Given these factors, it is somewhat surprising that the mean severity ratings were not more different. The absence of a correlation between respondents' severity ratings may be a result of the restricted range of severity ratings obtained. Virtually all referral concems were rated as being relatively severe, with 94% of teachers and 82% of psychologists rating severity at 5 or greater on a 7-point Likert scale.

Intervention Development

The high ratings for intervention planning involvement by both informants for themselves and their consultation partner indicated that intervention planning was jointly accomplished. This finding was consistent with the frequent emphasis on shared work and responsibility in the school-based consultation literature (e.g., Kratochwill et al., 1995). Respondents indicated that parents were involved in planning in slightly more than half of cases. Although the factors limiting parent involvement are not clarified by this study, the moderate level of parent involvement is a source of some

Dow

nloa

ded

by [

Uni

vers

ity o

f A

rizo

na]

at 1

2:29

17

Dec

embe

r 20

14


concern given how severe the teachers perceived the target concerns to be. The absence of a correlation between teacher and psychologist ratings of the consultant's involvement in planning may also be attributable to a restriction in range. Psychologists were rated as highly involved by both par- ties, and 85% of ratings from all respondents for psychologist involvement in planning were 5 or above on a '/-point Likert scale.

On average, teachers reported using one more element in the intervention than psychologists did. A considerable portion of this difference was attributable to differences in reporting the use of school-home notes. Twenty-two more teachers (30% of the sample) reported the use of school-home notes than did school psychologists. The general reporting of one additional component beyond those reported by school psychologists may also reflect teachers reporting activities they were engaged in on be- half of the student that were not planned as part of the intervention in consultation. Consultants may have been unaware of these additional teacher-developed activities or may not have perceived them as part of the intervention. Reporting an additional component also suggests that teachers may have perceived the intervention as more complex than school psychologists did.

Intervention Outcomes

Teachers and school psychologists reported similar moderate levels of student behavior change and satisfaction with the outcome of the intervention. This finding was generally consistent with previous findings regarding teacher perceptions of consultation (Gutkin, 1980). The infrequency with which school psychologists and teachers reported feeling accountable to either the student or parents for the outcome of the intervention is noteworthy. One variable that may play a role in the extent to which consultants and consultees feel accountable to parents is the degree of parental participation in consultation. The potential relation between parental participation and the perception of being accountable to parents was examined. For cases in which teachers reported that the parent was involved in planning or implementing the intervention, they reported feeling accountable to parents in only 42% of cases. Psychologists reported feeling accountable to parents in only 57% of the cases they identified as having parent involvement in intervention planning or implementation. Although both of these per- centages are higher than the percentage in the total sample, they are somewhat surprising given that these were the cases in which respondents re-

Dow

nloa

ded

by [

Uni

vers

ity o

f A

rizo

na]

at 1

2:29

17

Dec

embe

r 20

14


ported that parents played a role in intervention development, implementation, or both.

The extent to which consultees and consultants felt primarily responsi- ble to the teacher may also reflect the typical source of the referral concern in the context of these participants-the teacher. Although this study did not collect data to support this perception, it was a stated shared perception of the practitioners and administrators at this intermediate education agency that most consultation cases were initiated by teachers. The com- paratively modest degree to which consultants and consultees felt accountable for intervention effectiveness to the student and his or her parents raises concerns regarding the extent to which students are viewed as service recipients who are of similar importance to teachers (Noel1 & Gresham, 1993; Witt et al., 1996).

The finding that two of the variables that entered the stepwise regression for psychologists' perceptions of intervention implementation were ratings of teachers was consistent with previous findings (Martin &Curtis, 1981; Smith & Lyon, 1986) indicating that consultants attribute consultation's success or failure to consultees to a substantial degree. The negative relation between ratings of teacher involvement in planning and treatment integrity was unexpected and counterintuitive. To check the possibility that this was an artifact of teacher involvement entering the analysis after teacher knowledge, the correlation between psychologists' ratings of teacher involvement in planning and treatment integrity was examined and found to be r = -.055, p = .65. These findings were contrary to traditional recommendations for effective practice emphasizing the importance of teacher involvement in intervention planning (e.g., Bergan & Kratochwill, 1990; Gresham, 1989; Zins & Erchul, 1995). These data do not provide a basis for resolving the incongruent findings that school psychologists perceived teacher knowledge as positively related to treatment implementation but perceived teacher involvement in planning as unrelated to implementation.

It is interesting to note that for two of the three regression analyses (i.e., behavior change and satisfaction), psychologist's knowledge entered the equation. This finding indicates that teachers may attribute the success or failure of consultation to the consultant. This finding parallels previous studies indicating consultants attribute a consultation's outcome to the consultee (Martin & Curtis, 1981; Smith & Lyon, 1986). One possible explanation for the positive relation between consultant's knowledge and teacher perceptions of outcomes is that more knowledgeable school psychologists were able to help teachers develop more effective interventions. Another possibility is that teachers retrospectively

Dow

nloa

ded

by [

Uni

vers

ity o

f A

rizo

na]

at 1

2:29

17

Dec

embe

r 20

14


perceived psychologists as more knowledgeable if the students' behavior changed. Unfortunately, these data cannot resolve this issue. It is also interesting to note that teachers perceived preparation time as negatively related to treatment integrity and implementation time as negatively related to satisfaction. The negative relation between implementation time and satisfaction was consistent with previous findings regarding treatment acceptability (Elliott, 1988). The positive relation between perceived severity and perceived accuracy of implementation was also conceptually consistent with the acceptability literature (e.g., Elliott, Witt, Glavin, & Peterson, 1984).

Time Estimates

The preparation time estimates were substantially lower than those reported by Barnett et al. (1997). However, differences in the context (Head Start vs. public schools) and population (preschool-age vs. school-age children) may partially account for these differences. The difference in consultants (trainees vs. experienced certified practitioners) also may account for the differing time estimates. Additionally, differing time-cost estimates can be accounted for by procedural differences in the delivery of consultation. A noteworthy element of the time-cost data is that, on average, school psychologists estimated that they spent 31 min per week implementing intervention elements. This finding indicated that the psychologists participating in this study provided a significant amount of service in addition to the indirect service delivery that was traditionally emphasized in the consultation literature (e.g., Bergan & Kratochwill, 1990; Zins & Erchul, 1995). It is worth noting that psychologist implementation was distributed very un- evenly across cases: 63% of all psychologist intervention-implementation time was devoted to 15% of the cases. Thirty-two psychologists (43%) reported devoting no time to implementation.

Limitations and Future Research

Several limitations of this investigation are noteworthy. Most important, the data were the respondents' perceptions and not measures of their behavior. Although a self-report format is the most appropriate way to measure participants' perceptions and recollections, it does not inform the reader regarding the participants' actual behavior. Future studies of this type would be strengthened by collecting some direct measures of partici-

Dow

nloa

ded

by [

Uni

vers

ity o

f A

rizo

na]

at 1

2:29

17

Dec

embe

r 20

14


pants' behavior. A practical and logical next step for the study of naturally occurring services would be to collect intervention products such as intervention plans or outcome data. The addition of measures of behavior would provide an important additional anchor for comparison of perceptions and the correlates of perceptual agreement.

Two characteristics of the sample may limit the generality of the data. First, all of the participants were educators working in rural, semirural, and suburban school districts in the Midwest. To the extent that this specific context affects the problem-solving process of school-based consultation, these data may not generalize to other types of settings. However, the extent to which these macroscopic contextual variables affect consultation is not currently known. Resolution of this issue will require the collection of comparable data in differing contexts. Another limitation related to the sample is that the teachers' training in consultation, intervention, and team-based problem solving was not known. Although some information regarding school psychologists' training was available, this information was limited.

Two selection issues may have biased results. First, the return rate (43%), although similar to other consultation surveys (e.g., Sheridan & Steck, 1995), may have created some sampling bias in the final data set. A second potential source of sampling bias is the extent to which psychologists followed the survey directions. It is possible that some of the psychologists selected teachers with whom they had particularly enduring or effective working relationships, rather than the teacher for their last two completed cases. Although there was no obvious reason for psychologists to violate the instructions, given that the surveys were anonymous, com- pliance with instructions was not measured.

An additional limitation is a product of the focus of the study and result- ing design. This was a study of the perceptions of school psychologists and teachers involved in naturally occurring consultation cases that focused on the needs of individual children. As a result, the data analyses are descriptive and do not constitute tests of hypotheses. Hypothesis testing requires active manipulation of the consultation process and thus is incompatible with studying naturally occurring consultation cases. Additionally, the instructions for distribution of the surveys and the survey content focused participants on a specific referral for intervention for a specific student. These data do not address consultation targeting concems such as school-wide discipline practices or staff development. Finally, as with any survey instrument that employs forced-choice items (e.g., selecting primary concems and intervention elements), the choices presented to respondents may have limited or biased their responding.

Dow

nloa

ded

by [

Uni

vers

ity o

f A

rizo

na]

at 1

2:29

17

Dec

embe

r 20

14


Conclusions and Issues for Practice

These data were consistent with the consultation literature (see Sheridan et al., 1996) supporting the assertion that teachers and school psychologists perceive their consultations regarding children as an effective and satisfactory professional practice. This study added to that literature the specific finding that teachers and the school psychologists with whom they consult tend to develop similar perceptions of the student, consultation process, and intervention outcome. This process of developing a shared perception through working togther may contribute to the development of shared norms (e.g., Gutkin & Nemeth, 1997) and effective problem solving. For those variables where agreement was markedly lower, restriction in the range of ratings may have played a role in attenuating correlations.

The data also led to several concerns that warrant continuing attention in research and practice. First, consultation was largely limited to relatively severe concerns. To maximize the probability of consultation success and to allow consultation to serve as a form of secondary prevention, methods are needed that facilitate the initiation of consultatipn before problems represent long-standing, severe concerns. Second, the data for parent involvement were mixed with parents involved in approximately one half the cases. Methods are needed to increase parental participation in educational problem solving for their children (e.g., Sheridan, 1997), especially when the concerns are perceived as severe. Third, there appears to be a need to find ways to increase the status of students and parents in school-based consultation and the extent to which they are perceived as service recipients. A practical initial step might be to routinely include the student in some part of all consultation meetings. As is the case with any descriptive study, this investigation raises more questions than it answers. However, these data may help illuminate important considerations for practice as well as continuing research needs.

REFERENCES

Alderman, G. L., & Gimpel, G. A. (1996). The interaction between type of behavior problem and type of consultant: Teachers' preferences for professional assistance. Jourrial of Educa- tional and Psychological Consultation, 7,305-314.

Bamett, D. W., Bell, S. H., Bauer, A,, Lentz, F. E., Petrelli, S., Air, A., Hannum, L., Ehrhardt, K. E., Peters, C. A,, Barnhouse, L., Reifin, L. H., & Stollar,S. (1997). The Early Childhood Inter- vention Project: Building capacity for service delivery. School Psychology Quarterly, 12, 293-31 5.

Dow

nloa

ded

by [

Uni

vers

ity o

f A

rizo

na]

at 1

2:29

17

Dec

embe

r 20

14


Bergan, J. R., & Kratochwill, T. R. (1990). Behavioral consultation and therapy. New York: Plenum.

Costenbader, V., Swartz, J., & Petrix, L. (1992). Consultation in the schools: The relationship between preserv~ce training, perception of consultativeskills, and actual time spent in consultation. School Psychology Rez~iez[l, 21, 95-108.

Elliott, S. N. (1988). Acceptability of behavioral treatments in educational settings. In J. C. Witt, S. N. Elliott, & F. M. Gresham (Eds.), Handbookof behav~or therapy in education (pp. 121-150). New York: Plenum.

Elliott, S. N., Witt, J. C., Glavin, G., & Peterson,R. (1984). Children's ratingsof the acceptability of classroom interventions for misbehavior: Findings that influence teacher's decisions. lournal of School Psychology, 22,353-360.

Erchul, W. P., Hughs, J. N., Meyers, J., Hickman, J. A., & Braden, J. (1992). Dyadic agreement concerning the consultation process and its relationship to outcome. lournal ofEducationa1 and Psychological Consultation, 3,119-132.

Fuchs, D., & Fuchs, L. S. (1989). Exploring effective and efficient prereferral interventions: A component analysis of behavioral consultation. School ~ s ~ c h o j o g y Revinil, 18,260-279.

Gresham, F. M. (1989). Assessment of treatment integrity in school consultation and prereferral intervention. School Psychology Quarterly, 18,37-50.

Gresham, F. M., & Kendell, G. K. (1987). School consultation research: Methodological critique and future research directions. School Psychology Revieril, 16,3&316.

Gutkin, T. B. (1980). Teacher perceptionsof consultation services provided by school psychologists. Professional Psychology, 11, 637-642.

Gutkin, T. B., Henning-Stout, M., & Piersel, W. C. (1988). Impact of a district-wide behavioral consultation prereferral intervention service on patterns of school psychological service delivery. Professional School Psychology, 3,301-308.

Gutkin, T. B., & Nemeth, C. (1997). Selected factors impacting decision making in prereferral intervention and other school-based teams: Exploring the intersection between school and social psychology. Iournal of School Psychology, 35,195-216.

Ikeda, M. J., Tilly, W. D., Stumme, J., Volmer, L., & Allison, R. (1996). Agency-wide implementation of problem solving consultation: Foundations, current implementation, and future directions. School Psychology Quarterly, 11, 228-243.

Kratochwill, T. R., Elliott, S. N., & Rotto, P. C. (1995). School-based behavioral consultation. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology I11 (pp. 519-538). Silver Springs, MD: National Association of School Psychologists.

Lloyd, J. W., Kauffman, J. M., Landrum, T. J., & Roe, D. L. (1991). Why do teachers refer pupils for special education? An analysis of referral records. Exceptionality, 2,115-126.

Martin, R. P., & Curtis, M. (1981). Consultant's perceptions of causality for success and failure in consultation. ProfPssional Psychology, 12,67@676.

Noell, G. H., & Gresham, F. M. (1993). Functional outcome analysis: Do the benefits of consultation and prereferral intervention justify the costs? School Psychology Quarterly, 8, 200-226.

Noell, G. H., & Witt, J. C. (1996). A critical re-evaluation of five fundamental assumptions un- derlying behavioral consultation. School Psychology Quarterly, 11,189-203.

Reschly, D. J., & Wilson, M. S. (1995). School psychology practioners and faculty: 1986 to 1991-92-Trends in demographics, roles, satisfaction, and system reform. School Psychol- ogy Revieru, 24,62-80.

Robbins, J . R., &Gutkin,T. B. (1994). Consultee and client remedial and preventative outcomes following consultation: Some mixed empirical results and directions for future research. journal of Educational and Psychological Consultation, 5, 149-167.

Dow

nloa

ded

by [

Uni

vers

ity o

f A

rizo

na]

at 1

2:29

17

Dec

embe

r 20

14


Sheridan, S. (1997). Conceptual and empirical basesof conjoint behavioral consultation. School Psychology Quarterly, 12, 119-133.

Sheridan, S., & Steak, M. C. (1995). Acceptability of conjoint behavioral consultation: A national survey of school psychologists. School Psychology Reoiez~l, 24,63347.

Sheridan, S., Welch, M., & Orme, S. (1996). Is consultation effective? A review of outcome research. Retnedial and Special Education, 17,341-354.

Smith, D. K., & Lyon, M. A. (1986). School psychologists' attributions for success and failure in col~sultations with parents and teachers. Professional Psychology: Research and Practice, 17, 205-209.

Wade, S. E., Welch,M., & Jensen, J. B. (1994).Teacher receptivity tocollaboration: Levels of interest, types of concern, and school characteristics as variables contributing to successful implementation. 1ournal of Educational and Psychological Consultation, 5,177-209.

West, J . F., & Idol, L. (1987). School consultation (Part 1): An interdisciplinary perspective on theory, models, and research. journal of Learning Disabilities, 20,388408.

Witt, J.C., Gresham,F. M.,&Nocll,G. H. (1996). What's behavioral about behavioral consultation? journal of Educational and Psychological Consultation, 7,327-344.

Ysseldyke, 1. E., Christenson, S., Pianta, B., & Algozzine, B. (1983). An analysis of teachers' reasons and desired outcomes for students referred for psychoeducational assessment. lour- rlal of Psychoeducational Assessn~mt, 1,73-83.

Zins, J. E., & Erchul, W. P. (1995). School consultation. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology 111 (pp. 609424). Silver Spring, MD. National Association of School Psychologists.

George H. Noel1 is an assistant professor in the School Psychology Program in the Depart- ment of Psychology at Louisiana State University. His research interests include behavioral consultation, intervention implementation, and functional assessment.

Kristin A. Gansle is an assistant professor for research in the School of Social Work at Louisi- ana State University. Her research interests include treatment integrity, intervention, and social services for children and families.

Randy Allison is the supervisor of School Psychological Servicesat Heartland Area Education Agency 11 in Johnston, Iowa. He coordinates school psychology staff and services for 56 school districts in central Iowa. He has worked extensively on transformation of service delivery and staff development. D

ownl

oade

d by

[U

nive

rsity

of

Ari

zona

] at

12:

29 1

7 D

ecem

ber

2014

Documents

Do You See What I See? Teachers' and School Psychologists' Evaluations of Naturally Occurring