V A L I D I T Y OF P E R C E I V E D N E E D S A T I S F A C T I O N AS A C U R R I C U L U M E V A L U A T I O N C R I T E R I O N

Lindsay D. Mackay & Jeff R. Northfield

Purpose

The purpose of this paper is to report on investigations of the validity of one of the criteria being used in the formative evaluation of ASTEP materials: the extent to which the materials are perceived to satisfy certain student-teacher 'needs'. Before embarking on a description of the results, some of the background to the use of this criterion wil l be outlined.

Background: Satisfaction of perceived student-teacher needs as a criterion for STEP materials evaluation

Student-teacher motivation has been an important consideration in the development of materials for the Australian Science Teacher Education Project (ASTEP) and for its United Kingdom predecessor, the Science Teacher Education Project (STEP). The case for student moti- vation as a component in development of curriculum innovations and as a criterion for curriculum evaluation has been argued elsewhere (Haysom and Sutton, 1973/74, 1974), and wi l l not be repeated in detail here. In an attempt to establish motivational criteria for the evaluation of STEP materials, Haysom and Sutton derived a list of statements of student needs which they described as:

characteristics of the learner which can be harnessed to encourage a search for goals and objectives which might not have been immediately obvious to him.

(Haysom and Sutton 1973/74, p. 28)

This list was derived from tutor's reports on the reasons that student-teachers responded to, and became involved in, STEP activities, and from tape recordings of interviews with student-teachers on their reactions to STEP activities they had experienced. The way in which the list of statements was derived is described in detail elsewhere (Science Teacher Education Project, 1971 ; Haysom and Sutton, 1974).

The uses to which these statements of perceived student-teacher needs were put in the evaluation of STEP materials is described in detail by Holford (1974). Basically, student-teachers who had experienced a STEP activity were asked to rate their reaction to an activity and indicate whether or not each of the eleven needs statements were reasons that the activity appealed to them. Tutors were also asked to rate the extent to which the unit motivated student-teachers, and to indicate which of the list of eleven needs statements were met by the materials. This information from students and tutors was used in assessing the effectiveness of units.

The use of self perceived needs of student-teachers in the evaluation of A S T E P materials

Student needs have also occupied a central role in the development and evaluation of the ASTEP materials (Fensham, Northfield & Driscoll, 1974; Mackay, 1973a; Mackay & Northfield, 1974). Initial research conducted with the needs statements in Australia (Mackay, 1973b; Gunstone & Mackay, 1975) has indicated:

(a) that all but one of the eleven initial Statements of needs are consistently

114

interpreted, as far as meaning is concerned, by Australian student-teachers; (b) that the needs statements are ranked in a similar order of importance by

Australian and British student-teachers despite the diversity in attributes of the two groups (correlation between median ranking of statements by

the two groups = 0.6); and (c) that the perceived importance of the need statements changes during a

course of teacher-training.

On the basis of the experience of STEP and the acceptance of the STEP philosophy that student-teacher motivation and perceived satisfaction of the felt needs of student-teachers were important objectives for materials to be used in teacher education courses, perceived needs satis- faction was included as one of the criteria to be employed in the formative evaluation of ASTEP

materials. Data was collected from four sources:

(a) the author(s) of the unit were asked to specify the student needs which are met by that unit;

(b) student-teachers who had experienced an ASTEP unit were asked to indicate the extent to which they consider the unit contributed to the satisfaction of needs expressed in a number of statements on the three-point scale:

not at all or to a negligible extent,

to some extent,

to a substantial extent; (c) tutors who had used the unit in their course were asked to indicate the extent

to which they considered the unit contributed to the satisfaction o f student needs expressed in a number of statements on the same three-point scale;

(d) armchair critics who conducted an armchair evaluation of a unit were asked to indicate the extent to which they consider the unit is l ikely to contribute to the satisfaction of student needs expressed in a number of statements.

Of the lists of statements used to gather data from these four sources, ten statements were common to all four lists of need statements. These were:

1. To feel that you wil l cope more effectively in the classroom. 2. To feel you have achieved success (by your own standards). 3. To have new or fresh experience. 4. To sort out or express your own ideas or opinions. 5. To work with others. 6. To feel that your contribution is appreciated by others. 7. To see yourself more clearly as a teacher. 8. To feel you have improved your practical teaching. 9. To feel you have developed a better understanding of children.

10. To feel you have developed a better understanding of science.

Information on the extent to which the four groups of observers (authors, student-teach-

ers, armchair evaluators and tutors) perceived that these common statements of student-teacher needs were being satisfied, or were likely to be satisfied, by ASTEP units form the basis of this investigation.

115

Val id i ty of perceived needs satisfaction as a criterion for materials evaluation : the questions

The val id i ty of a criterion such as perceived needs satisfaction cannot be direct ly measured and expressed as a single index in the same way as the predictive val idi ty of an HSC examinat ion can be expressed in terms of its correlation wi th tert iary performance. Rather, the val idi ty must be established in terms of (i) the extent to which the informat ion on perceived

needs satisfaction is internally consistent and (ii) the extent to which the informat ion is consist-

ent with the initial construct of needs-satisfaction wi th in the student mot ivat ion model from which its use is derived.

Four questions were posed in an attempt to investigate the val idi ty of perceived needs

satisfaction as a criterion.

Question I. Are various ASTEP units perceived to satisfy different needs? If the cri terion does

not distinguish between units, then its usefulness as a cri terion is l imited. Figure 1 illustrates the needs perceived to be satisfied by student-teachers for four units with at least for ty student- teachers returns f rom at least two institutions. In this figure, the eleven needs statements are grouped into four groups of needs based on previous analyses.

Persona/Needs:

4. To sort out or express your own ideas or opinions. 3. To have new or fresh experience. 2. To feel you have achieved success (by your own standards).

Social Needs:

5. To work wi th others. 6. To feel that your contr ibut ion is appreciated by others.

Coping as a Teacher Needs:

1. To feel that you wi l l cope more effectively in the classroom. 8. To feel you have improved your practical teaching. 7. To see yourself more clearly as a teacher.

Understanding Needs:

9. To feel you have developed a better understanding of children. 10. To feel you have developed a better understanding of science.

1 1. To feel you have developed a better understanding of science education.

It is apparent from Figure 1 that di f ferent patterns of needs are perceived to be satisfied

by various units. Thus the criterion of perceived needs satisfaction does discriminate between ASTEP units and can be used to distinguish between units.

Question 2. Are the needs perceived by student-teachers to be satisfied by ASTEP units those which the authors specify as met by the units? An answer to this question can be obtained by comparing the mean percentage of student-teachers who perceive that a need is satisfied by a unit for two sets of needs -

�9 those nominated by the authors as met by the unit; and �9 the other needs.

One would anticipate that a higher percentage of student-teachers would indicate that the needs

116

o 1 t I ' v

"~~176 ~ \

*%% ~ %

f f

" ~ / / J ~ .-- .-- s 1 1

/ ,.N"

\ / i t J / / / / �9 /

/ / ' / / /

�9 / " / l /" / ]

�9 ". / . . . . .

l I I I {DI I I I I 0 0 0 0

Z ~ o~

z~ O. ~. O0 UJ

W Z

:D

Z o,=,, 0 . . 2 : O r . ) U <

U.,l l--

~_j~ Z

Cl uJ I ~ I ~ o uJ lJ

I 0

C

Q . LIJ l - l /) <

j-,

.S o

!

| Z . .

I.Ll

LI.

117

m ~

~._~.

�9 ~ @

g ~ m

o I

\ \

t ~ . - n

" o 0 O O m

0

m m ~ Z

,-t

I I I I 1 I I !

\ \

. \ §

01 ~ 0

-I-

\

Z ,-n ,-0 O m m

z . - n m ~, .-I Z

O if) ~ n ,

_ _ m

~ O Z C Z " l

m

m I

~ s

\

118

nominated by the authors were satisfied by the unit than would be the case for other needs. This is so for nine of the twelve units with more than ten student-teacher evaluation sheets available as indicated in Figure 2. For five of the units a Mann-Whitney U test indicates that the difference in the anticipated direction is significant beyond the (x = .05 level. A Kolmogorov-Smirnov test applied to the values of U obtained for the whole set of twelve units indicates that there is a tendency for needs nominated by authors as met to be more frequently reported by student teachers to be satisfied by the unit than for other needs.

Question 3. Do student-teachers, armchair evaluators and tutors consistently report a unit as satisfying the same needs? This analysis was restricted to analysis of the data on six units which had a satisfactory amount of data for two or three groups of reporters on the unit, and for which the data happened to be readily available at Monash.

In order to answer this question, Spearman Rank-order correlation coefficients were computed for each unit between the percentage of respondents reporting a need as satisfied for various groups of reporters. The results are summarised in Table 1.

TABLE 1

Spearman Rank-order correlations between percentage of respondents reporting various needs as satisfied by an ASTEP unit

for three groups of respondents

Unit

73/05 73/08 73/34 73/35 73/01 73/07

Correlation between student-teachers

and armchair critics

.50

.41

.06

.39

.74

Correlation between student-teachers

and tutors

.33

.14

.83

.44

.76

Correlation between armchair critics

and tutors

.68

.61

.37

.86

Median value .41 .44 .64

For all three comparisons of pairs of respondents in Table 1, the set of correlations is such that a Kolmogorov-Smirnov test indicates that there is a significant correlation (beyond (~ = .05) between the percentages of respondents in the three groups reporting needs as satisfied by ASTEP units.

Question 4. Is a student's overall rating o f a unit related to his~her perceptions of the extent to which the unit contributes to satisfaction of specified needs? At the simplest level, one can test the proposition that the more needs a student-teacher perceives as satisfied, the more highly he or she wil l rate the unit. This corresponds to the "needs-richness" notion proposed by Haysom & Sutton (1974).

The data for 158 student-teachers who reported on 10 of the ASTEP units is summarised in Figure 3. There is a positive correlation (r = .12, p < .07) between number of needs reported as

1 1 9

o

c

m t~ g~ ~

0

0

S t u d e n t - t e a c h e r ra t ing o f u n i t

_=..

0

I

I )< i

I X i

I X I

i #, i

I X i

I Y,

I ) ( I

==

0

I

0

120

satisfied and how interesting the student-teacher found the unit. This indicates that the number of needs a student-teacher reported as satisfied by a unit could account for about 1.4% of variance in his or her rating of the unit.

At a more sophisticated level, one can establish a model which allows for different con- tributions to the rating of a unit for satisfaction of different needs and for different levels of perceived satisfaction (to some extent and to a substantial extent). If one carries out an analysis based on such a model, 19% of the variance in student-teacher ratings of a unit can be accounted for.

As anticipated, when a need was perceived as satisfied to a substantial extent it made a greater mean contribution to rating of the unit. In addition, if one takes the mean contribution that satisfaction of a need makes to the rating of a unit as an indication of the perceived import- ance of that need, this can be compared to the data on the reported importance of these needs for a large sample of student-teachers obtained by Gunstone and Mackay (1975). There was a Spearman rank correlation of .738 between the reported importance of these needs statements and the mean contribution that perceived satisfaction to a substantial extent of this need made to the rating of that unit.

Summary and conclusion

In summary, then, the results reported in this paper provide clear support for the validity of perceived need satisfaction as a criterion for evaluation of ASTEP materials.

Specifically it has been demonstrated that:

�9 different ASTEP units are perceived to satisfy different student-teacher needs; �9 authors of ASTEP units, student-teachers, tutors and armchair critics exhibit

consistent perceptions of the needs satisfied by particular ASTEP units; and �9 perceived satisfaction of needs is related to the student-teacher's rating of that

unit in terms of its interest, and the more important the need is perceived to be, the greater the contribution that its satisfaction appears to make to the rating of the unit.

Having demonstrated that perceived needs satisfaction is a valid criterion for the evaluation of teacher education materials, this raises the question of whether perceived needs satisfaction could or should be a criterion for evaluation of other curriculum materials.

References

FENSHAM, P., NORTHFIELD, J. & DRISCOLL, D. "ASTEP - A model for developing curriculum materials in teacher education." South Pacific Journal of Teacher Education, 2(1): 5-11, 1974.

GUNSTONE, R.F. & MACKAY, L.D. "The self perceived needs of student teachers." South Pacific Journal of Teacher Education, 3 (1) : 44-51, 1975.

HAYSOM, J.T. & SUTTON, C.R. "Motivat ion: a neglected component in models for curriculum improvement." Curriculum Theory Network, 4 (1): 23-35, 1973/4.

HAYSOM, J.T. & SUTTON, C.R. Innovation in Teacher Education. McGraw Hill, London, 1974. HOLFORD, D.G. "Evaluation: A process of distill ing experience" and "Case studies in the evalu-

ation of particular curriculum units", in Haysom, J.T. and Sutton, C.R. Innovation in Teacher Education. McGraw Hill, London, 1974, 77-90 and 94-109.

121

MACKAY, L.D. "A proposal for the formative evaluation of ASTEP materials." Mimeographed Report, ASTEP, Canberra, 1973(a).

MACKAY, L.D. "ASTEP interim research report no. 1/73." Mimeographed Report, ASTEP, Canberra, 1973(b).

MACKAY, L.D. & NORTHFIELD, J.R. "The use of a formative evaluation model for ASTEP." Research in Science Education, 4: 65-75, 1974.

SCIENCE TEACHER EDUCATION PROJECT. '~Vith student teachers in mind." Information Bulletin No. 3, 1971 (mimeo).