INSTRUMENTS FOR EVALUATING SCHOOL PSYCHOLOGISTS’ FUNCTIONING AND SERVICE

JONATHAN SANDOVAL NADINE M . LAMRERT

Iiniversity of Califoniia, Davis University of California, Berkeley

The evaluation of school psychological services is both a responsibility and an opportunity too long ignored. Unique problems must be overcome in evaluating school psychologists, but the task is not insurmountable. This paper describes and presents examples of five data collection devices which can be used in the evaluation effort: the vignette-based questionnaire, the role-model questionnaire, the services received questionnaire, the teacher interview, and nonobtrusive measurement.

The purpose of this paper is to discuss the process of evaluation and to present some instruments that may be used in evaluative research. We have not attempted to review efforts a t evaluation which have been reported in the literature, but rather have attended to common problems and instruments which may be adapted to a variety of purposes. There is little necessity to elaborate on the need for the evaluation of school psychological services. All professional educators must prove that their efforts are cost effective in a time of decreasing resources and increasing community pressure for verifiable educational outcomes. School psychologists cannot be exempt from this process and must be accountable to the public. At the same time, they must subject themselves to self-evaluation and professional growth to meet the challenge of social change. Evaluation plays a most important part in providing the feedback to guide professional developmental efforts.

It is ironic that school psychologists have not set the pace in educational evaluation, since they have a greater range of research skills and broader knowl- edge of evaluation than any other employee in most school districts. There are many possible explanations for the relative invisibility of school psychologists on the evaluation scene. Since most professionals develop a self-concept in con- nection with their work, few wish t o undergo the emotional turmoil which accom- panies a close, objective evaluation of what they are doing. Of course, too, an evaluation is time-consuming, and psychologists are busy enough without extra tasks of this sort. Another reason for evaluative inactivity, however, may be the lack of instrumentation available to conduct self-examination of functioning.

EVALUATION PRACTICES AND PROBLEMS THAT MAKE SCHOOL PSYCHOLOGISTS VULNERABLE

School psychologists are ubiquitously evaluated by tallies of their various activities. Counts are made of such things as number of tests given each month, number and types of people seen, number of children screened for special education, number of reports, number of administrative conferences, number of in-service hours, and so forth. Beyond these frequency counts, districts evaluate psycholo- gists via the principal’s or administrator’s ratings of the quality of the psychol- ogist’s work including such areas as punctuality, appearance, and friendliness.

Requests for reprints should be sent to Jonathan Sandoval, Dept. of Education, University of California-Davis, Davis, CA 95616.

172

Instruments for Evaluating School Psychologists 173

The problem with frequency counts of work activities is that, while it may indi- cate if the psychologist is doing something or nothing, i t provides no information about the quality of the work performed. Further, it does not inform the psycholo- gist as to which of his many activities are paying off and which are not. Ratings by school administrators, of course, have limited utility because administrators are often only peripherally involved in most psychologists’ activities, although they are a valuable source of some information. Nevertheless, because work load statistics and ratings are quick and easy to collect, they continue to be used.

In the ideal case, the psychologist should be evaluated by the impact of his services on the children of a school. However, since his efforts are often far removed from the child and since most of the time he must work through a teacher, an aide, or an administrator to effect any change in the child’s life, perhaps it is more practi- cal to evaluate the impact he has on a teacher. Focusing on the teacher is risky, however. Psychologists are often called in for assistance as a last resort after a child has accumulated a long history of educational failure with its accompanying behavioral consequences. When they are unable to conjure up an immediate magical cure, they are rejected by teachers for failure to meet (unrealistic) expectations. Then, too, there is built-in resistance on the part of teachers to almost anyone in the school system with higher status and pay. If the psychologist is able to assist a teacher, i t is by helping that teacher to be more effective, using his own resources along with any information the psychologist may offer. Since the teacher must do most of the work with the child by himself, the psychologist’s contribution is soon forgotten or never recognized. Evaluation is even more complicated when the psychologist spends time working to prevent children from failing. If the effort is successful, the result is children performing normally in school, and this success can be demonstrated only by actuarial figures collected over a long time. We are much more aware of illness, for example, than we are of good health, which we take for granted and attribute to many causes.

In spite of these drawbacks, teachers are the best source of information about psychological services. Efforts directed a t enlisting their opinions about the psy- chologist’s performance may also serve to educate them to alternative kinds of services available. In the authors’ efforts to evaluate the functioning of intern school psychologists, we have produced several teacher-oriented devices which have potential applicability for other evaluations. Before describing them we would like to comment on one methodological problem in evaluation.

EVALUATION DEVICES WHICH CAN YIELD USEFUL INFORMATION ABOUT PSYCHOLOGICAL SERVICE

Sample Consideration A compromise which often must be made in an in-school evaluation of a psy-

chologist concerns the sample. In conducting an evaluation, i t is seldom convenient to draw an ideal random sample of the teachers in the schools where a given psy- chologist works. Psychologists are thinly distributed in school systems and i t is next to impossible for them to make contact with all of the teachers in all of the schools to which they are assigned. Then, too, some teachers choose not to use the services of the psychologist for one reason or another, as is their right. One

174 Psychology in the Schools, A p r i l , 197’7, Vol. 14, No. 2.

option in creating a reasonable sample is to select randomly from those teachers with whom the psychologist has had some contact. Either the psychologist may list ones with whom he has worked, or an administrator may search school records to learn the identity of these teachers. A psychologist may have a great deal of contact with one teacher but work only for a short time with another. A good idea would be to stratify the sample on the basis of the amount and type of collaboration that has existed between the psychologist and the teacher. It may well be that the teachers whom a psychologist nominates as the ones with whom he has worked extensively constitute a biased sample; nevertheless there is value in knowing how helpful the psychologist has been with those teachers. If he has been successful, he may continue in his efforts, attempting to extend them to others. If he has not been successful with those whom he, himself, selects as the ones he works with most, he had better examine his efforts most thoroughly.

Vignette-Based Questionnaire‘ An indirect way to assess teachers’ perceptions of the usefulness of school

psychologists is to learn what sorts of children they refer and how frequently they would be apt to refer them to a school psychologist; i.e., with what kinds of cases do they see the psychologist as being helpful. In a vignette-based questionnaire, all of the school consultants including the principal are listed beneath a brief case history description of a particular type of child. The teacher’s task is to indicate which of the resources he would be likely to call upon for assistance. Cases clearly appropriate and clearly inappropriate for referral to a school psychologist may be included to test the limits of the teacher’s confidence in the psychologist’s help. The assumption underlying this method is, of course, that teachers will continue to request assistance from psychologists who have succesfully aided them in the past. Appendix 1 presents a sample of a questionnaire of this type and a number of vignettes.

The data presented in Appendix 1 come from U.C. Berkeley evaluations of intern school psychologists. The results are teachers’ rank orderings of the listed school personnel. They are summarized as proportions of ranks in various cate- gories, because different school districts may have varying numbers of school resource personnel to be ranked, making averages of ranks inappropriate. I n addition to the problem of summarizing the ranks, this ranking procedure forces raters to assign the personnel to equal intervals, implying that the magnitude of preference for one referral source over another is the same for all adjacent ranked pairs. While this sort of scale may be useful for some purposes, the procedure provides more information if the raters space their ratings of the importance of a resource on a continuum from 1 to 100, with behavioral labels describing resource utility anchoring various positions on the scale. On the continuum, 05, for instance, might be indicated as signifying ((never consider as a source of help;” 25 might signify “sometimes useful in this situation if other help has failed;” and 95 might indicate “always would seek this individual out for help, for he or she has been consistently useful.” Two relatively positively perceived personnel might receive ratings of 95 and 90, while three other less frequently sought after resources could

‘Reginald Corder of the Educational Testing Service has assisted in the development of many of the instruments reported.

Instruments for Evaluating School Psychologists 175

be rated 50, 20, and 05. Averaging these ratings across judges provides excellent data for evaluating teacher perceived effectiveness of several resources.

Role Model Questionnaire This type of questionnaire has been described in a previous report (Lambert,

Sandoval, & Corder, 1975). We have considered it important to attempt to identify some of the global impressions teachers have of school psychologists, i.e., how they perceive them in general. In this questionnaire, various descriptions of school psychologist are offered, and the teacher may choose the one description which best fits the psychologist in question. Appendix 2 presents the general descriptions the authors have used in their evaluation efforts.

Services Received Questionnaire A services received questionnaire is a way of presenting the teacher with a

list of all possible psychologist activities or all of those which have been provided. The teacher may be asked a number of questions about the activities, such as: Does your school psychologist provide this service? Has he provided this service to you in the last year? Would you like the psychologist to provide more of this service? Would you like your psychologist to do less of this activity? This type of questionnaire has the advantage of informing the teacher as to what service can be expected from him, as well as assisting the psychologist to view the reaction of individuals to specific things he is doing in the school. Appendix 3 contains a list of possible items for this kind of questionnaire.

In evaluating the responses to teacher questionniares, it is important to realize that they represent the crystallization of teacher perceptions. These perceptions can and should be contrasted with the psychologist’s own perceptions of his func- tioning. If the psychologist also completes the questionnaire and reflects his own efforts, discrepancies in the results can become a source of dialogue between teacher and psychologist, where more effective ways of working together may be planned.

Teacher Interview Questionnaires as data collection devices suffer many disadvantages (Kerlinger,

1965). One way to improve on the effectiveness of questionnaires is to turn them into interviews. Although they are time-consuming and costly, a skilled interviewer may produce information of an invaluable sort. We have used the services received questionnaire as a structured interview with great success. The data are the same as those reported above, as well as the spontaneous comments of teachers regarding psychological service needs and perceptions.

Nonobtrusive Measures AS we pointed out earlier, procedures based on teacher self-reports are practical

and economical for conducting evaluative research. Nonobtrusive measurement provides another means by which to obtain evaluation data. Nonobtrusive or nonreactive measures are used when there is a desire to collect information without sensitizing the person being measured to the process (see Webb, Campbell, Schwartz, & Sechrest, 1966). School people are no different from others and feel un- easy when asked to evaluate someone. To escape the discomfort of having to rate someone, many will hurriedly check an indicator in the average range just to avoid

176 Psychology in the Schools, April , 1977, Vol. 14, No. 2.

value dilemmas. Unobtrusive measures get around this problem by counting al- ready existing data or occurrences. The following sources of information might reveal interesting information about psychologists’ functioning:

1.

2.

3. 4.

5 .

6. 7.

The attendance of the teachers on the day the psychologist is regularly a t the school. The number of people who sit with the psychologist a t lunch or during recess. The number of teachers who greet the psychologist in the halls. The variety of referrals from teachers, i.e., do they see the psychologist as being helpful with more than one type of child? The number and type of teacher requests for help which are not generated by a referral. The number and type of requests from school principals and parents. The cost of service for child for whom either direct or indirect services are provided.

REFERENCES

KERLINGER, F. N. Foutidations of behavioral research. New York: Holt, Rinehart & Winston, 1965. LAMBERT, N. M., SANDOVAL, J., & CORDER, R. Teacher perceptions of school-based consultants.

Professional Psychology, 1975, 6, 204216. WEBB, E. J., CAMPBELL, D. T., SCHWARTZ, R. D., & SECHRPST, L. Unobtrusive measures: Nonre-

active research i n the social sciences. Chicago: Rand McNally, 1966.

APPENDIX 1 T H E VIGNETTE-BASED QUESTIONNAIRE

Situation # 1 - A first-grade boy has difficulty keeping his seat for any period of time. He does not follow directions unless they are given to him directly on a one-to-one baqis. This situation tends to disrupt your classroom agenda far too often and has become personally exhausting to you.

Proport,ion of Ranks Assigned: Teacher Referral Source High Middle Low Respondents

School Psychologist .72 .19 I09 57 Guidance Counselor . 56 .26 .18 23 Child Welfare and Attendance

Worker .29 .14 .57 7 School Nurse Speech Therapist

.32 .44 .24 5 7

.09 .21 .70 23 Curriculum Consultant . l l .15 .74 27 Principal .40 .32 .28 57

Other situations used as stimuli: Situation # 2 - The sixth-grade girls are known to be smoking in the irls’ rest room. Your

talks with them have not produced results. To whom do you turn for help. Situation # 3 - A sixth-grade boy has been persistently absent from school. You suspect that

the notes he brings excusing his absence have been written by someone other than the parent. You have not been able t o discuss the problem with the parents.

Situation # 4 - Your district has been notified that it will receive a large amount of federal aid (money) to conduct a stepped-up program for culturally disadvantaged children. Each teacher is

f

Instruments for Evaluating School Psychologists 177

expected to plan a curriculum and prepare a budget for those disadvantaged children who will be in his/her room.

Situation # 5 - This fourth-grade boy is unable to read first-grade materials or do simple compu- tations in addition and subtraction. His general appearance, carriage, bodily movements, and speech suggest to you that he may be mentally retarded.

Situation # 6 - This child appears a t school with bruises on her arms and rings under her eyes in addition to being generally tired and irritable with the other children. When questioned about her home, she doesn’t answer directly.

Equal Interval Format

-Always would seek this individual out, for he/she has been con- 100

90 sistently useful. 80-Has been helpful in similar situations in the past. 70 60 50t-Would never fail to inform about the situation for would probably

40 30

have some valuable input.

-Sometimes useful in situation like this if other help has failed.

-Almost never helpful in a situation such aa this. 20

10 0-Never useful in this area.

APPENIIIX 2 ITEMS FOR A ROLE MODEL QUESTIONNAIRE

(Counseling Model)

(Case St,udy Model)

(School-Based Consultation

Model)

(Special Education Psychological Examiner )

(Counseling Model)

# 1

# 2

# 3

# 4

# 5

This school psychologist is employed primarily to assist the teaching staff by therapeutic counseling of individual upils or groups of pupils about their social and emotional problems. Stufents are frequently counseled over an extended period of time. The results of the counseling int,erviews are reported in writ,ten form at the end of the series of interviews. This school psychologist is employed primarily to assist the teaching staff by making studies of individual children whose learning problems require t,he use of s ecial methods of appraisal or study in order to be understood by the teacKer. A major share of the psychologist’s responsibility is discharged by making studies in depth of children and writing reports which indicate the findings of t.he studies. This school psychologist is employed to assist teachers in resolving student problems by working with teachers to develop an understanding of such problems. While the psychologist’s ultimate goal is to help the child in school, he does not assume the responsibility for outlining a program of action. Using information from many sources, including his own t.est,ing, the psychologist collaborates with the teacher in determining what activities or procedures t.he particular teacher may employ to assist the child a t the time of difficulty. The psychologist and teacher jointly determine the next steps to take and share the responsibilit#y for the decision. This school psychologist is employed primarily to screen candidat,es for special classes for t,he gifted, emotionally disturbed, mentally retarded, etc. After giving a battery of tests, the psychologist submits a report of his findings to ot,her persons who decide what to do about placing the pupils in special classes. This school psychologist, is employed primarily to assist teachers and pupils to find solutions for social and emotional problems. Students may be counseled individually or in groups and over extended periods of time. Without be- traying the confidential nat,ure of the counseling interviews, the psychologist brings general findings from these interviews to teachers in an effort t,o de- velop a “ t.herapeutic atmosphere” in the classroom.

178

(School-Based Consultation

Model)

(Case Study Model)

Psychology in the Schools, April , 1977, Vol. 14, No. 2.

# 6 This school psychologist is employed to help teachers and administrators anticipate pupils’ problems, as well as remediate them. He routinely meets with teachers to discuss children in the classroom who are having difficulty. He helps the teacher to perceive the children from a different point of view through bringing information from the psychologist’s classroom observations, interviews, and testing of referred pupils to his meetings with the teacher.

# 7 This school sychologist is employed primarily to assist the teaching staff by making stuges of individual children whose learning r b l e m s require the use of special methods of study or appraisal in order to e understood by the teacher. These studies are made in considerable depth and the psychologist uses a variety of instruments for measuring intelligence and personality. In addition to writing a report which indicates the findings of his study, the psy- chologist outlines suggested procedures for the remediation of the child’s problems and gives some idea of the child’s probable course of future behavior.

APPENDIX 3 [TEMS FOR THE SERVICES RECEIVED QUESTIONNAIRE

SCHOOL PSYCHOLOGIST SERVICES :

I. Assessing planning and evaluating a program for an individual child A. Consulting with me about. a child with a problem:

I . 2.

3.

4.

0.

6. 7

Provided information to assist in determining the nature of the child’s problem. Helped me to examine the child’s present behaviors and the classroom settings in which thev occur. Helped me to specify the learning or behavior goals which were appropriate for the child. Determined whether the child could perform the learning task or comply with the behavior requirements. Added to my knowledge of individual differences of learning and behavior charac- teristics of children. Determined what teacher responses were rewarding or satisfying to the child. Determined what factors affected the child’s motivation in school

B. Working with me to plan an individual education program for a child: 1. 2. 3.

4.

5 . Aiding me to evaluate an educational program for a child: 1. 2. 3 . 4.

Discussed optimal learning materials and classroom settings for a child. Helped me to find materials appropriate to the child’s level of performance. Discussed with me available school resources and ways of using them to assist an individual child. Planned methods of increasing and maintaining the child’s motivation for or interest in learning. Helped me to plan ways to involve the child’s parents in his educational program.

Aided me to evaluate and revise previous educational plans for a child. Helped me to find ways to assess a child’s abilities and disabilities. Assisted in developing a sequence of steps necessary for a child’s successful learning. Helped me to plan ways to determine whether a child was progressing toward specific educational goals.

C.

11. Assessing and evaluating programs for groups of children in my class A. Consulting with me about the characteristics of children in the class:

1.

2.

3.

4.

Working with me to plan a program for the pupils in my class: 1. 2.

Provided information to help me assess the learning and/or behavior characteristics of the children in the classroom. Helped me to specify teaching objectives for my classroom or a group of children within the class. Assisted in determining whether the children in the group were capable of achieving the teaching objectives. Helped me develop realistic expectations of my professional impact on the children in my class.

Helped me understand better the range of individual differences in the class. Assisted me to plan a program to achieve objectives.

B.

Instrumen,ts for Evaluating School Psychologists 179

3. Provided information about materials and methods which might be useful. 4. Helped me to evaluate the progress of children in the classroom toward specific edu-

cational goals. ,i. Assisted me in developing techniques to manage disruptive behavior in the classroom

when it does occur, and to prevent such behavior from occurring in the future. Working with teachers and other school staff in planning, executing, and evaluating special educational projects for children: 1 . Helped to assess learning and behavior characteristics of children for whom a special

program was t>o be planned. 2. Assisted in specifying behavioral objectives for a program. 3. Helped to determine what measures would be employed to assess change. 4. Entered into the planning of the educational program. 5. Cooperated in determining special materials or methods for carrying out. program goals. 6. Participated in the activities of the special pogram. 7. Assumed responsibility for some phase of program evaluation. 8. Reported results of special program to me and other teachers.

C.

111. Developing and planning a program to obtain Special Education Service for children A.

B.

Planning and developing programs withiti the regular classroom setting for the children with learning problems: Planning and developing programs including special materials, individual tutoring, or a special placement in additiot, to that of the regular classroom: 1. Remedial reading program. 2. Miller-Unruh early childhood program. 3 . Intellectually gift,ed program. 4. Speech and language program. 5 . Visually handicapped program. 6. Physically handicapped program. 7. Acoustically handicapped (moderate). 8. Learning Disabilities Group program. Providing information to help determine whether a child is eligible for these special pro- gram placements: 1. 2. 3 . Home teaching program. 4. Orthopedically handicapped class. 5 . Physically handicapped class. 6. Gifted program.

C.

Educationally handicapped (Special Day Class). hlentally handicapped (EhIR or TMIt).

IV. Providing in-service education experiences A.

H.

C. 1).

Increased my knowledge about human development, learning theory, group processes, and methods of assessment and observations applicable to the classroom. Assisted me to develop parent conference methods which were effective in gaining parent cooperation. Helped me to devise a research plan to investigate some educational question. Provided more knowledge about how services of school psychologists might be used in plan- ning and evaluating programs for children.

V. Conferring with parents A. I3. C. I ) . E.

Aided me in planning conferences. Participated with me in a parent conference. Helped me t.o utilize information from parent conferences. Recommended a referral or other professional service for a child. Helped me to find roles in the classroom for parents who wish to become involved and offered assistance in planning training sessions for volunteers. (Note: Similar to IB5).